Big decisions

This has been the first week of coding, officially. And it has been a surprisingly tough one. One of the major design decisions has been made, I have a somewhat bigger understanding of Geeklog plugin development and I feel a slight sense of accomplishment.

Baby steps

The first thing was to make something, a good place to start (if you are interested in making a plugin for Geeklog) is here. This is a detailed handbook. If however you are like me and like to see things examining the plugins which ship with Geeklog is a good approach. I use the Polls plugin as my reference as it has most resemblance with what I am doing. Last but definitely not least, the thing that gave me a kick start is the Plugin Toolkit. It is useful, after providing some basic data about your plugin it will generate a lot of code for you. After that you have to divide the generated files to the right places and of course edit what you need. Again it is a good start it will not develop a plugin.

So what I did was use the plugin to generate some files. After that things went south for a bit. First of all was the decision on who is allowed to do translations. The community decided that logged in users, with credentials provided by the central Geeklog "page" would be allowed to translate. This discusion also included a discussion about how the database should look like and well the final form is as suggested by Wim Niemans. Creating a database table with Geeklog is as simple as editing a sql file (a sample file is generated by the toolkit).
After that I played a bit with the configuration. It is not final it is not even a large step towards the final it was more a exercise for me than providing actual code. But after doing it I am ready to do some real damage when I get to that part. My approach so far was to build simple parts, working good enough to allow me to tackle what I though would be the "big" problem.... And a big problem it was.

The big bad LANG

Before I get to the problem, let me describe how Geeklog core works.
Geeklog has it's language folder with language files, e.g. english.php, bosnian.php, german.php...
A snipet from the english.php file:

$LANG01 = array(
    1 => 'Contributed by:',
    2 => 'read more',
    3 => 'comments',
    4 => 'Edit',
    5 => '',
    6 => '',
    7 => '',
    8 => '',
    9 => 'Admin Functions:',
    10 => 'Submissions',
    11 => 'Stories',
    12 => 'Blocks',
    13 => 'Topics',
    14 => '',
    15 => '',
    16 => '',
    17 => 'Users',
    18 => 'SQL Query',
    19 => 'Log Out',
    20 => 'User Information:',
    21 => 'Username',
    22 => 'User ID',
    23 => 'Security Level',
    24 => 'Anonymous',
...

There are a lot more arrays with a lot more element. The elements can be one word elements or multiple sentences. They can include html code and so forth and so on. They also include common words such as "Save" which can be otherwise included in the page, as a user comment, as a page generated by the static pages plugin... So the big problem is identifying the strings on a rendered page. This is important as the whole goal of the plugin is to provide a context for the translation.

There were several possibilities for solving this. My first idea was to add HTML tags to the array elements such as <span class="translator">.....</span> but the problem with this was that these spans can not be included in places such as the <title></title> or inside a <input value="">. Another approach was to add a API to Geeklog, although this would probably work perfect, and without much hiccups I wanted to avoid doing changes to the Geeklog core.

My (currently) final solution
I went on with the idea of adding some kind of identification to the array elements. Spent some time trying out special HTML symbols such as &#8204. This symbol is invisible for the user, which made it perfect it also was invisible to JavaScript which made it useless.
So I moved on to adding simple text. What I finally have is that all the elements (in the default language file) have these marks: _-start_ and _-end_. I tried to devise something unlikely to appear otherwise on the page.
This change is done when the plugin is installed and is removed if and when the plugin is removed. It will (in the near future) also provide a way to add the identification to other language files (if by some chance the default language of the page is changed).
After doing this, in order to save computation time later I also added data on which array and which element the current string belongs to.
The english.php file now looks like this:

$LANG01 = array(
    1 => '_-start_||array=$LANG01index=1||Contributed by:_-end_',
    2 => '_-start_||array=$LANG01index=2||read more_-end_',
    3 => '_-start_||array=$LANG01index=3||comments_-end_',
    4 => '_-start_||array=$LANG01index=4||Edit_-end_',
    5 => '',
    6 => '',
    7 => '',
    8 => '',
    9 => '_-start_||array=$LANG01index=9||Admin Functions:_-end_',
    10 => '_-start_||array=$LANG01index=10||Submissions_-end_',
    11 => '_-start_||array=$LANG01index=11||Stories_-end_',
    12 => '_-start_||array=$LANG01index=12||Blocks_-end_',
    13 => '_-start_||array=$LANG01index=13||Topics_-end_',
    14 => '',
    15 => '',
    16 => '',
    17 => '_-start_||array=$LANG01index=17||Users_-end_',
    18 => '_-start_||array=$LANG01index=18||SQL Query_-end_',
    19 => '_-start_||array=$LANG01index=19||Log Out_-end_',
    20 => '_-start_||array=$LANG01index=20||User Information:_-end_',
    21 => '_-start_||array=$LANG01index=21||Username_-end_',
    22 => '_-start_||array=$LANG01index=22||User ID_-end_',
    23 => '_-start_||array=$LANG01index=23||Security Level_-end_',
    24 => '_-start_||array=$LANG01index=24||Anonymous_-end_',
...

I am not saying that this is the best possible solution. I am saying that it works and it does not demand changing the coding style of Geeklog. If you have a better solution leave a comment, write me an email, write an email to Geeklog-Devel mailing list...

Doing just this (of-course) makes the page ugly, very ugly.
About this much:

That is why I added some JavaScript magic, to remove the marks and to extract the data I will need to create the translation interface.

This is a sample object I get:
string: Advanced Search, array: $LANG01, index: 75

And this is how the page looks after JavaScript:

Note that not all the strings will be red all the time. The plan is that the user can pick the string he wants to translate and that the picked string will be highlighted. The highlight might be changing the color, or increasing the font size or all of it.

That is the big step I made towards completing this project.

Again If you have any suggestions please feel free to contact me.

For the Geeklog community: You Rock!!! And thank you for having patience with me.

Thanks for reading.

Just before KickOff

This blog is the place  I will use to gradually report what I have done during the summer. While I will probably skip information about my lunch I will post on the progress of my Google Summer of Code project.

Lets start with a nice introduction on the organization. I think the official website has a better intro than I could write so:
"Geeklog is an open source application for managing dynamic web content. It is written in PHP and supports MySQL, PostgreSQL, or MS SQL as the database backend.

"Out of the box", Geeklog is a CMS, or a blog engine with support for comments,trackbacks, multiple syndication formats, spam protection, and all the other vital features of such a system.

The core Geeklog distribution can easily be extended by the many community developed plugins and other add-ons to radically alter its functionality. Available plugins include forums, image galleries, and many more."

Geeklog sadly did not get into GSoC officialy this year, but thanks to The Fedora Project they still got one spot. Which btw makes me feel a whole lot of pressure.

"Fedora is a fast, stable, and powerful operating system for everyday use built by a worldwide community of friends. It's completely free to use, study, and share."

There is a lot more to learn about both Fedora and Geeklog that I copied here, and I would encourage You to do so.

I hope that the next blog entry will contain more blocks of code than words, but this is the community bounding period, in which to be honest I did not offer my best performance. In my defense however it is the Final exam period on my university and I did have 4 exams in 4 days,with one more pending.

Well I did not write a whole lot but I have been reading the mailing list, actually several of them. What I have learned is that the Geeklog community is really great and patient. And as this project does frighten me from time to time the patient part is very important. After all we, or I am a student,bound to make a few mistakes now and there. Not to broadcast the wrong idea here, I do have cold feet but I am looking forward to having a lot of fun, learning and hopefully producing a great plugin this summer.

My project
In one sentence: "A plug-in for the Geeklog CMS which would enable users to submit translations".
A really detailed explanation of the project is inside my proposal.
It will try to build a plug-in which will allow the CMS users to do translations for the CMS. It should help maintain and improve the current 30+ translations Geeklog's UI has and add some new ones to the mix.

It will also allow users to make their own, private translation, I am not sure how much Geeklog would use this but if You want you would be able to go ahead do a translation to say Vulcan.

What I did so far

I am laying down the groundwork, under the guidance of Dirk Haun, the mobile presenter .

First I got my very own code repository, which I look forward to filling. Then I had to get down with the basics of Mercurial. To make my life a bit easier I decided to use TortoiseHg for as a interaction tool with the repo.

The next step for me was to make sure  I have all the right tools on all the OS running on my laptop. OK that sounded a bit ambiguous, I have 3 systems so it is not quite the hard work but still...

I got done some reading on the Geeklog wiki, and probably will do a lot more. But If You are new to Geeklog and think it is worth Your time this might be a good place to start.

During the day, or tomorrow I will go ahead and make my own minimal plugin, probably something easy and funny just to get started. (Change all the occurrences of "CMS" on a page to "sandwich" ?)

So my setup for the summer is Windows 7/Ubuntu 12.04.02/Fedora 18 with Wamp/Xampp and sublime text 2. And of course Google for when I hit a great brick wall.

Again looking forward to a great summer. And I hope to justify the trust I got from Geeklog.

Unrelated stuff

I don't remember this happening last year but this year the GSoC mailing list got a thread about unsatisfied students. While I do understand the need for feedback and the unpleasantness of not getting accepted I do think I it is not the best thing to publicly write about how the chosen student sucks in comparison to the rejected. (No I have not been mentioned but still..) I do not claim to know on what the exact selection procedure is,but logic implies that the org will pick someone they see more fit to complete the job-given it is in the org's interest to have the job done.

How does pushData works

On a previous post I have shown how to use push data.
Since then there have been some changes to it and as for the user end they are shown here: http://www.youtube.com/watch?v=ZEf6JRb5OF8&feature=youtu.be
That video might not be the best one you watched but it is just a draft, final version will come soon enough.
However this post is not about how to make a new procedure, but what happens in the background.

When a new procedure is created
Obviously the app creates a whole lot of panels, labels, buttons and lists. For the most part they are the same as any other app would do.
Here I want to show one part of it (this is used in a few places)

1  var folder=DocsList.getFileById(ss.getId()).getParents()[0];2  var files=folder.getFilesByType("spreadsheet");3  for(var i=0; i<files.length; i++)4     sourceSpreadsheet.addItem(files[i].getName(), files[i].getId());

This piece of code is responsible for the list boxes where the user will choose source and destination spreadsheet.

The list has a on change handler which updates the list of sheets in that spreadsheet.



The rest of the user interface is quit simple. The real work starts when the user hits the finish button.

In the first steps all the user-provided data is mapped from various text and list boxes.

 1 var newPush={}; 2    3    4   newPush.sourceSpreadsheetName=sourceSS.getName(); 5   newPush.sourceSpreadsheetId=sourceSS.getId(); 6   newPush.sourceSheetName=sourceSheetName; 7   newPush.headers=headers; 8   newPush.destinationSpreadsheetName=destinationSS.getName(); 9   newPush.destinationSpreadsheetId=destinationSS.getId();10   newPush.destinationSheet=destinationSheetName;11   newPush.row=startRow;12   newPush.column=startColumn;13   newPush.time=timeInterval;14   newPush.keepEmpty=e.parameter.keepEmpty;15   var id=1;16   while(db.query({type:"pushData", pushId:id}).hasNext()==true)17     id++;18   newPush.pushId=id;19   newPush.lastPush=new Date().toString();20   newPush.success="Done";21   newPush.type="pushData";22   var realTime=false
23       if(timeInterval==0){24         newPush.realTime="true";25         realTime=true;26       }27   else
28     newPush.realTime="false";

After all the data is collected the app makes a new object and saves the provided data in it

Most of this is just saving data, however lines 15-17 might seem a bit odd.

ScriptDb has an integrated function getSize() for query results, but from what I have seen getSize() sometimes returns false results.

That is why the id is calculated manually here.



The next part is to create the trigger that will track changes on the source spreadsheet.

And this was not as easy as it sounds. At the time of writing the app there was no way to determine the source of the onEdit trigger. Yes, there is a e.source for onEdit triggers

but that will return the app’s host spreadsheet, not the spreadsheet where the change occurred. There I used a little trick…

The trick is (and it is not very elegant) I have 10 hardcoded onEdit functions (called editOn)

 1 function editOn1(){ 2    3   markEdit("editOn1"); 4    5 } 6 
 7 function editOn2(){ 8    9   markEdit("editOn2");10   11 }12 
13 function editOn3(){14   15   markEdit("editOn3");16   17 }18 
19 function editOn4(){20   21   markEdit("editOn4");22   23 }24 
25 function editOn5(){26   27   markEdit("editOn5");28   29 }30 
31 function editOn6(){32   33   markEdit("editOn6");34   35 }36 
37 function editOn7(){38   39   markEdit("editOn7");40   41 }42 
43 function editOn8(){44   45   markEdit("editOn8");46   47 }48 
49 function editOn9(){50   51   markEdit("editOn9");52   53 }54 
55 function editOn10(){56   57   markEdit("editOn10");58   59 }60 
61 function markEdit(root){62   63   var db=ScriptDb.getMyDb();64   65   var results=db.query({type:"editHandler", editFunction:root});66   67   var obj=results.next();68   obj.haveChanged="true"; 69   db.save(obj);70   71   72 }

As You can see all of them have one line of code, they call the “markEdit” function sending as a parameter the function name.

And this is why…

…As I had no way to find out which spreadsheet was edited but wanted to avoid unnecessary database updates at all cost I made a little on edit web.

When a user defines a new process the source spreadsheet gets an onEdit handler assigned (that is IF it does not have one previously assigned).

When a spreadsheet is changed the onEdit (that is editOn) function will fire, and record the change to ScriptDb. Later on, when the app has to update the database

it will only update sheets that are assigned to the function which recorded the change. (the code is bellow in the part about the update of db)

After the tedious process of checking if the source spreadsheet has a onEdit handler then assigning it… Or if the push interval is set to real time assigning the real time trigger the app will go ahead, save the sheet to ScriptDb and finally perform the first data push.



Updating the database

As is stated in the post Big data sets and ScriptDb I had to assume a lot of spreadsheets with a lot of data and I had to find an alternative way of saving the data.

This solution works, and No I am not 100% happy with it and I intend to change it soon enough (working on a library special for cases such as pushData) but the app had to be released and this is what we have now.

 1 function updateProces(){ 2    3   var db=ScriptDb.getMyDb(); 4    5   var changedSS=db.query({type:"editHandler", haveChanged:"true"}); 6    7    8   while(changedSS.hasNext()){ 9     10     var editObj=changedSS.next();11     12     var spreadsheets=Utilities.jsonParse(editObj.spreadsheets);13     14     for(var k=0; k<spreadsheets.length; k++){15       16       var ss=SpreadsheetApp.openById(spreadsheets[k]);     17       var sheets=ss.getSheets();18       19       for(var l=0; l<sheets.length; l++){20         21         var sheet=sheets[l];       22         var hasProcedure=db.query({type:"pushData", sourceSheetName:sheet.getName()});23         if(hasProcedure.hasNext()==true)24           updateDbSingleSheet(ss, sheet)25             26           }27     }28     29   }30   31 }32 

Remember the object edited in markEdit, well here the app will query for all of those objects who’s haveChanged attribute is set to true.

That is it will generate a list of all spreadsheets that are changed (at least potentially changed). Again I had no way of finding the actual source of the change so I work with arrays of spreadsheets.

All the spreadsheets associated with the trigger the fired onEdit are remapped in the database. updateDbSingleeSheet has been documented in the post bellow Big data sets and ScriptDb (hack)



Pushing the data

Another trick here, only data that has been changed will be pushed. And again some of it has not been actually changed but is associated with the onEdit handler which recorded a change.

First the app will generate a list of changed spreadsheets.

1 while(changedSS.hasNext()){2     3     var editObj=changedSS.next(); 4     mergeArray(listOfChangedSS,Utilities.jsonParse(editObj.spreadsheets));5   }6   

Then it will cross reference that list with procedures who's push time interval has expired( or plainly it will check if it is time to push the data)

1 if( ( (timeStart-lastPush)/3600000)+0.25>=obj.time){
2       if(underscoreGS._contains(listOfChangedSS, obj.sourceSpreadsheetId)==true)3         objectsToPush.push(obj);4     }

underscoreGS is a awesome library with a bunch of functions for manipulating arrays. (it can be found in the Google Apps Script documentation in the part Notable Libraries)

The app will then go through the list of objects to push, and do the push, sheet by sheet.

 1 unction writteData(headers, sourceSheet, destinationSheet, sourceSpreadsheet, row, column, keepEmpty){ 2    3   var db=ScriptDb.getMyDb();   4   var items=db.query({itemType:"sheetData", spreadsheetId:sourceSpreadsheet.getId(), sheetName:sourceSheet}).sortBy("itemId", db.NUMERIC); 5    6   var completeData=[]; 7   for(var i=0; i<headers.length; i++){ 8      9     completeData[i]=[];10     completeData[i][0]=headers[i];11   }12   13   while(items.hasNext()){14     15     var thisItem=items.next();16     for(var i=0; i<headers.length; i++){17       if(!completeData[i])18         completeData[i]=[];19       20       mergeArray(completeData[i], Utilities.jsonParse(thisItem[headers[i]]));21     }22     23   }24   25   completeData=ArrayLib.transpose(completeData);26   27   if(keepEmpty=="false")28     completeData=findEmpty(completeData);29   30   var mR=destinationSheet.getMaxRows();31   var lR=destinationSheet.getLastRow();32   if(lR>completeData.length)33     destinationSheet.getRange(row, column, lR, completeData[0].length).clear();34   destinationSheet.getRange(row,column,completeData.length, completeData[0].length).setValues(completeData); 35 }36 

The arguments of this function is what the user provided while creating the process. Headers-headers to be pushed, row- the row where the data writing should start, column- the column where the data writing should start, keepEmpty-is weather the empty rows from the source sheet should be kept in the destination sheet or filed with the next available non-empty row.

First thing the function does is assemble all the pieces of data from ScriptDb (while saving the data it might be divided into smaller pieces)

Then it will clear the columns in which the data will be written. (why? well if you had 100 entries previously and now have only 50 this will delete the remaining rows)

Finally the data is written and the app will proceed to the next process.

pushData

The best way to explain what pushData does is by the words of the maker of original pushData , Andrew Stillman.

Here it is a nice picture explaining it:

 

Andrew has been my mentor on all the projects I did this summer and a very good and thorough teacher.

pushData was his idea and out there is a published version, the difference between the previous version and this is basically that here the user gets a graphical interface, this version uses ScriptDb and as suggested by users of the previous version there is a real time transfer option included.

 

How it looks

All parts of the look are not final, but here it is.

After installing the app the onOpen handler will add an additional menu to your spreadsheet:

The above entries will do:

Start pushData – will show the user interface where you can define new processes, edit/delete existing processes do an immediate data push (overriding the time interval) and see all processes defined

About pushData - will display the picture first shown in this post

Update database now – will check for changes in your source sheets and update the database accordingly

Clear all data and triggers – will delete whatever you and the script made and give you a fresh start

Package push procedures for others to copy – will generate Google Apps Script code for you that you can share. Say you have a great system of processes and want to share it with someone, this is what you need.

Import copy procedures – will use the code generated by the function above to initialize the processes. NOTE: the spreadsheet names have to bee the same as from the original user

Manage user tracker settings – it is great to allow user tracking, the tracking will not allow anyone to steal the users identity but will help New Vision.

 

Start Screen

The grid will provide data on any active processes, Push all data now button will provide an immediate push and New Push Procedure will get the user to the interface for defining a new procedure.

 

The starting point of creating a new procedure is to pick the source spreadsheet, afterwards the app will allow for picking the source sheet. After the source sheet has been picked the app will provide a grid with an overview of the first 5 rows in the source sheet.

 

The next step is to chose the headers to be pushed. The app provides a set of list boxes, every box will contain all the headers from the source sheet. The order in which the headers are picked is the order they will appear in the destination sheet.

After the headers have been picked the user is prompted to choose the destination spreadsheet and sheet. It will also allow for picking the starting point for the data to be written. A range check will be done, if there is a possibility for overwriting data in the destination sheet the user will get a warning. A grid will provide a overview of data currently in the destination sheet.

 

Finally the user will pick the push interval, options go from 1 to 24 hours for all processes, but will allow real time pushing for one process.

After hitting the finish button the app will save the source sheet to ScriptDb, and preform the first data push taking the user back to the start screen,

 

pushData is most definitely the best app I produced this summer of several reasons. On of the main is that Andrew took a very active role in developing this.

Additionally the previous version of pushData has received some feedback which are implemented in the new version.

 

Cheers