It's the final countdown

This project could logically be divided into 3 major parts:

• Phrases recognition
• Translations interface and awards
• Packing

This final stage is about packing, although I have already written both code and blogs about packing this is about something else. This is about sending data from one website to another. This was challenging every step of the way, but I managed to get something I like; And with time to spare. Google Summer of Code is approaching it's end. Today is 'Soft Pencils Down' which means that students have just about 7 days left to finalize projects, write documentation , work out those little bugs... I have to say, on one side I am glad it will soon be over, those who know me know that I have been working on a few projects besides Geeklog and it took a lot of energy out of me, so I welcome the break, just a few days. I say a few days because my classes start soon and I have no intention of leaving the CrowdTranslator as it is now, or as it will be in a few days.

Credentials

The first idea for remote submission was to allow all sites using Geeklog to submit to Geeklog's master page. However when I started working I decided to make it more crowd and web like. Basically in what I have now any site using Geeklog CMS can submit translations to any other site using Geeklog. of course it would be plain stupid to allow just anyone to send data to a database, and given that Geeklog is the secure CMS I have a feeling someone would have shot me if I submitted such code. What I taught was a good solution was to allow admins to give credentials to pages which are allowed to submit. The interface for this is very simple (take a look at the pic). The creation of the account is also very simple but only because I borrowed some magic from Geeklogs user registration.

Basically if you have 2 sites, lets call one Geeklog and the other Loggeek , Geeklog's admin will go to this page and create a username and password for Loggeek and give this username and password to the Loggeek admin. Later on Loggek's admin can send translations to Geeklog via a very similar form which looks like the picture right. In order to send data Loggeek's admin will have to provide the URL to Geeklog, the username and password(credentials) provided.

I have managed to re-use a portion of the local translation submission code for the remote submission - very good thing.

Finally after the translations are submitted the user is provided with a thank you screen. And I threw in a little extra which I intend to work on later on - bragging iframe. At this point the iframe simply states something like:

 We have submitted 19 translations to www.ius-students.com (ius-students.com is a web domain I had laying around so I use it for testing of the plugin). What I intend to have here is a collection of badges just like any user would get. The page generates the code so Loggeek's admin can just copy it and paste it in somewhere in the page.

The rules

I have also set up a set of preliminary rules for submissions which in shot are:

• Only translations with more than 1 positive vote are sent
• Translations which are already sent once will not be sent again
• You need to get credentials from the site you are submitting to

Tweaking the admin index

With adding the new functionality to my plugin I had to add a bit to the admin page. So now the admin page will also display the total number of remotely submitted translations. It will also display the translations and the site names in the translations table.

lib-translator.php

This file was supposed to be the library for the plugin,now it is exactly that. 90% of the plugin code is located there. The reason for this- mainly security, +Dirk Haun  suggested that this way we have less potential attack vectors, the other reason I love how it looks. Although I could not get myself to place all the code there some of it just logically does not belong in there. If however it turns out that it would be majorly better I will do it.

User profile

I also added the personal translations list to the user profile now.

Next step

The next step is writing documentation, it does not like me I don't like it but we ll get along somehow.  Meanwhile Dirk will probably find a bug or two in my code so fixing this is mandatory. After that I will hopefully be able to say that I successfully completed two Google Summer of Code programs, am accepted into a new awesome community and will get my 5 days of blissful ignorance.

Until the next reading,

Cheers

P.S.

For anyone wanting to see a demo of the plugin feel free to make an account on www.ius-students.com and play with it. The admin credentials are as provided by Geeklog.

Translation in the jar

The next logical step in the Geeklog adventure was to pack submitted translations. They would not be of much use in a database would they?
So I spend the past days working on a script to pack everything up.

Geeklog has it's standard for language files, this I had to 'copy' into the new files. I also had to re-assemble the phrases with their HTML tags and PHP variables, and finally make sure that the variables are submitted as exactly that a variable, not the calculated value of it.

Quote type

First of all, Geeklog s strings vary between single (') and double (") quoted. What I needed to do in order to have the string back as they were was to get which quote type is used on which translation. As I anyhow had to get the names of all the arrays (LANG01,LANG03,.... $MESSAGE,...) and that implies traversing the language line by line I included the quote type finding into this code. It all looks as bellow.

The next part was using the retrieved language array names to get their actual values, the values are (in Geeklog ) stored in the $GLOBALS variable, so I made use of that. This seamed like the best possible approach to my problem.



After this is preformed the translation as actually packed in the system's language folder.The function print_array will do a lot of string manipulation in order to create the file output. It will place the quotes, replace variable values with their names and retrieve translations from the database. Any non existent translation will simply be replaced by it's English counterpart.

Of course it was not as simple as it seems here, there were quite a few tricks in the packing process, one of the first I noticed was that I got the string "Don\' t have..."(this is fixed now). There were no life changing problems or solutions but it was challenging enough to keep me busy.

59 => "Don't have an account yet?  Sign up as a <a href=\"{$_CONF['site_url']}/users.php?mode=new\" rel=\"nofollow\">New User</a>",

Cheers

Highlight 'em all

So it has been a while now that I switched from marking to mapping.

The new approach has it's ups and downs of course, the ups are that it is crazy and it works, the downs are... everything else. Lucky for me, everything else is fixable :D

First of all I want to point out this thing: http://beta.phpformatter.com/ it is the best formatter I found out there, has just enough options, and makes the code much more readable. If you have a lot of nested conditional statements or loops it is a life saver as it adds comments after each } on what the conditional is.

Saving the original

I have, in detail, explained the saving of the original phrases here. The previous approach worked, but had a few bugs or rather it relied on certain formatting paradigms. I have updated this so now it uses actual values instead of file manipulation.
The trade-of here is that the plugin can not be installed while Geeklog is installing. Since the chances that this plugin will ever ship with Geeklog this is not a big problem.

The trick is that the LANG array names are still parsed from the language file, however the variables used (I have checked and additional variables are not likely to be added) are redefined , instead of the real value the variable gets the variable name e.g.
$_CONF[ 'site_url' ] = "{\$_CONF['site_url']}";

So when the value is parsed I still get what I need. In order not to render the page useless (at least before a reload) the original variable values have to be returned.

The main reason I re-implemented this (so close to deadline) is because I worked on the packing of the translations. The same logic will be used there, but with a small intermediate step - if there is a translation for a certain phrase it is retrieved from the database and used, otherwise the English phrase is used.

Highlighting

The biggest problem with the new approach was highlighting the string on the page. Of course for phrases such as "Directory" the implementation could be trivial. However Geeklog phrases are not that simple, there are variables, there is HTML and so on and so forth.




I have tried a lot of implementations for this, and they all failed. Until I realized that the perfect implementation was something which already exists. It is naive , it will highlight "Contributed" when searching for "Contribute" but it is the best implementation I could use.

It relies on execCommand which will flatten any HTML between string parts, all I had to do was to make sure that the search string has no <var> or <tag> tags the plugin uses, which was easy enough to do.

Finally the plugin is getting where it need to be, by week end I hope to have finished the packing functionality and therefore completing a minimalistic implementation. After that, there is a list of things I was asked to added to increase the usability of the plugin.


Cheers

Language markup goes to Language mapping

The midterm evaluation at GSoC passed and I am proud to say that I passed. The passing rate is around 94% so one might think this it is not a big deal... Well, we are a bunch of dedicated and smart people :)

However the midterm evaluation made us (mostly +Dirk Haun ) do a thorough review of the code and it revealed a few bugs which were easy to patch. However it also revealed a infinite loop. When I sat down to fix this I realized that the problem is a bit deeper then a infinite loop. After we sat down to think about it, the language markup (read here) revealed more and more problems.

The problems

Geeklog does some string shortening on some pages, e.g. "This text would be shorter" in rendered as this "This text". This caused the infinite loop, the language markup algorithm relies on having pairs of _-start_ and _-end_ tags. The natural approach to fix this would maybe be to find all the pairs of _-start_ and _-end_ and just ignore any "string" which has another _-start_ before an end tag. The natural approach would fail because Geeklog has nested strings. (When I write strings here I mean strings from $LANG variables).

The second problem we got was that not all Internet users have JavaScript enabled. Before concluding that this is a small number of users etc etc, lets make one thing clear, I am thinking of web crawlers. They (as far as I know) do not have JavaScript enabled and you don't want your page to be represented as 

_-start_Another nifty Geeklog site_-end_.

The final problem was that in some cases element (such as forms) ids are set to be LANG arrays, after the page is rendered and my JavaScript kicks in the complete page is purged of all language markups => The ids of the forms would not match the expected ids in the PHP script.

All of these problems might even be fixed, but they were bound be reincarnated later on in some way.

The solution

+Dirk Haun and I have bounced ideas left and right for a few days trying to figure out a new solution. At one point I even suggested an API (desperate times). However a somewhat better (and crazier) idea came to mind and I have been on it ever since.

The solution is language_mapper.php. The logic behind it: When the plugin is installed it will traverse Geeklog's file tree and find all the .php files (for now we are avoiding plugin folders). I am going with a recursive depth first approach for this.

After all the files have been found they are "analyzed" , the code will compile an array holding all the actual LANG array names and search for them in the file's code. It will save the list of found array names as well as a list of included (required) files.


All of this is saved to the database.
Later on when a page is loaded the plugin's JS file will send an AJAX request and get the form HTML. Most of this cod stayed the same, in fact generally speaking most of the code stayed the same. I listened to what smarter people than me said and used a certain level of abstraction with my code. It was a bit hard or unachievable in some places but I did it well enough that most of the code simply worked with the "new data" provided to it. This made me happy as I don't have to redo all of it, and probably +Dirk Haun as we were not moved back to square one.

Why save the included files?

The included files (obviously ) contribute to the script in some way, as far as I know that contribution could be in form of text or HTML code, so in order to really assemble a list of LANG's used on a page I have to include the one's used on included pages.

The problem with this

The problem with this approach is that in coding we have all those conditionals, if, switch .... This means that not ALL the code will be executed ALL the time. So I end up with a list of 349 LANG elements out of which 149 have been actually displayed on the page. In order to keep the 'in context' translation it is sort of necessary to remove the overhead. I have wrestled with this for the better part of last night but all I got was a very slow JS function. My guess is that it is slow because of the nature of the strings, they are not fixed. They have variables in them so I have to use regex matching. Another problem might be that most of the search terms are not present on the page so the complete page has to be searched

In conclusion

Although there is a problem with the approach it has been said "long ago" that this has no "nice solution". I like how it behaves for the most part and will try to fix this "small" inconvenience.

Cheers