Language markup goes to Language mapping

The midterm evaluation at GSoC passed and I am proud to say that I passed. The passing rate is around 94% so one might think this it is not a big deal... Well, we are a bunch of dedicated and smart people :)

However the midterm evaluation made us (mostly +Dirk Haun ) do a thorough review of the code and it revealed a few bugs which were easy to patch. However it also revealed a infinite loop. When I sat down to fix this I realized that the problem is a bit deeper then a infinite loop. After we sat down to think about it, the language markup (read here) revealed more and more problems.

The problems

Geeklog does some string shortening on some pages, e.g. "This text would be shorter" in rendered as this "This text". This caused the infinite loop, the language markup algorithm relies on having pairs of _-start_ and _-end_ tags. The natural approach to fix this would maybe be to find all the pairs of _-start_ and _-end_ and just ignore any "string" which has another _-start_ before an end tag. The natural approach would fail because Geeklog has nested strings. (When I write strings here I mean strings from $LANG variables).

The second problem we got was that not all Internet users have JavaScript enabled. Before concluding that this is a small number of users etc etc, lets make one thing clear, I am thinking of web crawlers. They (as far as I know) do not have JavaScript enabled and you don't want your page to be represented as 

_-start_Another nifty Geeklog site_-end_.

The final problem was that in some cases element (such as forms) ids are set to be LANG arrays, after the page is rendered and my JavaScript kicks in the complete page is purged of all language markups => The ids of the forms would not match the expected ids in the PHP script.

All of these problems might even be fixed, but they were bound be reincarnated later on in some way.

The solution

+Dirk Haun and I have bounced ideas left and right for a few days trying to figure out a new solution. At one point I even suggested an API (desperate times). However a somewhat better (and crazier) idea came to mind and I have been on it ever since.

The solution is language_mapper.php. The logic behind it: When the plugin is installed it will traverse Geeklog's file tree and find all the .php files (for now we are avoiding plugin folders). I am going with a recursive depth first approach for this.

After all the files have been found they are "analyzed" , the code will compile an array holding all the actual LANG array names and search for them in the file's code. It will save the list of found array names as well as a list of included (required) files.


All of this is saved to the database.
Later on when a page is loaded the plugin's JS file will send an AJAX request and get the form HTML. Most of this cod stayed the same, in fact generally speaking most of the code stayed the same. I listened to what smarter people than me said and used a certain level of abstraction with my code. It was a bit hard or unachievable in some places but I did it well enough that most of the code simply worked with the "new data" provided to it. This made me happy as I don't have to redo all of it, and probably +Dirk Haun as we were not moved back to square one.

Why save the included files?

The included files (obviously ) contribute to the script in some way, as far as I know that contribution could be in form of text or HTML code, so in order to really assemble a list of LANG's used on a page I have to include the one's used on included pages.

The problem with this

The problem with this approach is that in coding we have all those conditionals, if, switch .... This means that not ALL the code will be executed ALL the time. So I end up with a list of 349 LANG elements out of which 149 have been actually displayed on the page. In order to keep the 'in context' translation it is sort of necessary to remove the overhead. I have wrestled with this for the better part of last night but all I got was a very slow JS function. My guess is that it is slow because of the nature of the strings, they are not fixed. They have variables in them so I have to use regex matching. Another problem might be that most of the search terms are not present on the page so the complete page has to be searched

In conclusion

Although there is a problem with the approach it has been said "long ago" that this has no "nice solution". I like how it behaves for the most part and will try to fix this "small" inconvenience.

Cheers

Gamification, because hard work should be awarded

The time came to add gamification to the mix. The most part of it was research, gamification as powerful as it is gamification does have pitfalls. If someone reading this ever has to do some sort of gamification implementation I found these to be really good:

• NY Times
• Trans Media
• iMedia Connection
• Academic Earth

Of course the first resource should be your own thinking - How would I cheat this system.

The other problem was making some sort of scalable code, and since we don't have a full list of awards as for now it is hard to find the common denominator. In fact for now I have 2 very different awards. The first one is 'First translation' (as the name implies it is given after the user submits their first translation) the other one is 'n-th translation'. The huge difference is that one is given only once, while the other is 'given' or rather updated several times.

The implementation

So I started with 2 functions I knew I would need, one which will do the connection of user and 'gem' - give the user his award. The other which will check if the award has been previously given. This is also the first line of defense, you have the award you are not getting it again. The second line of defense is that if you do try to submit crappy input the admin will probably block you and if you are blocked all the translations, all the votes and all the badges are deleted - Crime and Punishment.

The next step was to check what the award possibilities are. All the awards or gems are saved in a DB table gems, this is filled with data upon installation of the plugin (or rather will be). When a user submits translations the gems in the gems table are taken, the previously awarded gems are subtracted and a list of possible awards is formed.

All the repetitive gems are added to the list via PHP. It just occurred to me that a better way might be to add an additional column to the gems table indicating if the gem can be awarded more than once...

With the list in hand, or array in memory , the code is of to checking if the user has meet criteria to receive awards. The 'First Translation' was easy, the criteria for 'n-th translation' was the problem.

What is n?

And here is where it got a bit tricky. The awards where to be given on a 'regular' basis, so for every n translations you should get acknowledged. The step size had to be increasing. Since we talk about cc 1700 translations a user can submit, per language, I had no intention to hard code all the milestones. There is where math comes in handy. I was off to a Google search (2 awesome pictures at the bottom) to find some nice numerical series with increasing step size , but where the step size does not get too big too fast.
The answer for me was Pentagonal number series. The first few steps are 1, 5, 12, 22, 35, 51, 70, 92, 117, ... and the general rule (3n^2-n)/2.

Another award will be given for n-th week in a row, but this should be basic counting, and for first vote. Both will be implemented during the day.

The interface update

So giving awards is a nice feature, one of the suggestions I got on Geeklog's mailing list was to display all the possible badges to the users, which is work in progress. But the first thing which came to mind was to notify the user that he got something, this however is not made persistent as for now, and I am not sure if ti will be.

The other update to the plugin is it's connection with Geeklog's user profile.
Again as for so many things in the known universe Geeklog has an API for this, all it requires is to put a function named plugin_profileblocksdisplay_<plugin_name>($user_id) in the functions.inc file and return the HTML of the code to be added.

Another request from the mailing list is to allow users to see all the possible badges, so I did a quick update to the function in lib-translator.php which does the badge display. The user profile is as above, but the Crowd Translator public page now looks like this:

There it is the first step, or few small steps towards gamification of the translation process. The community is was helpful, I got a whole load of suggestions and tips, probably not possible but I will try to implement all of them.
And last night while writing the first part of the blog I got a nice surprise, the mailing list had a new entry, of course I went to see what it is and found this:
BTW Thanks for your hard work.

I don't expect it, it is my privilege and pleasure to do this plugin but it sure is nice to hear.

And the 2 awesome pictures:

Plugin Admin Page

In some logical order of things (at least in my head) the next step after making the Plugins index page was making the Plugins admin page. Before I got on to that I had to change bit's and pieces all over the code. First of all I realized why the $_TABLES['table_name'] trick did not work, mainly because I did not add my tables to the list. /*feels ashamed*/. After fixing this I had to go on and find ALL the sqls used int the code and fix this. Additionally +Dirk Haun reminded me that I am missing some basic security for my lib-translator.php. As indicated before lib-translator.php will at the end be the collection of all the functions the plugin uses. So the security , the first defense line is this piece of code:

If you are interested in what the $function part is just go here.

Admin Page is not Index Page

Why I have to specify that the pages are not the same? Well because for a good part they do look alike.
Here is how the admin page could look like.

Lets break this page down to pieces to show what the differences are.

Quick Stats

On the Index page the user had preview of his own stats (and some for the plugin). Here it is all about general Plugin stats, on this particular screen shot the data is such:

Quick Stats

Translations submitted:	150
Total votes:	150
Most up votes:	100
Users translating:	2
Languages being translated:	2
Translations with negative vote count:	0

Basically it is self explanatory the nice part is the last row, the number of translations with negative vote count (approval count). At this point let me explain how approval count is generated. Every translation starts with a approval count of 1 (ap=1), the assumption is that the translation submitter would vote for his own translation. Every time a user hits the   the ap is increased by one, every time a user hits the ap is decreased by one. (In the case a user changes his vote from +1 to -1 ap=ap-2  or the other way around)

After the little detour, the last row in the quick stats indicates the number of negative votes, now the plugin will (said it here) delete translations automatically once they reach a certain negative approval count (e.g. -5), but here the admin gets a heads up so he can do something about the translation submitter. More about this later on.

The next part is the status and list of languages being translated. One thing missing here is the "Pack Language" button, this will be added later on and basically will pack the translation to a format Geeklog can use. 

Badges

The Index page displayed the current user's badges, the Admin page gives a view of all available badges. I put this here because the plan is to allow the page admin to change the badge image. The badges will be displayed on the user's profile(as for so many things in the known universe Geeklog has an API for that), and I wanted to give them a possibility to adjust the look of the bade to the look of the page. On some page, someone who submitted a lot of good translations could get the badge:

The table

About this part I am proud. I wanted the table not only to be a place where translations could be viewed I wanted it to actually be useful. So here the table is 

User (ASC)	Language(ASC)	Translation	Upvotes (ASC)	Posted (ASC)
test_user1(block)	Fake Input	DaqmeynzhG	93	2 hours ago	delete
test_user1(block)	Fake Input	tkrLqNmeiL	96	2 hours ago	delete
test_user1(block)	Fake Input	woXsaXTNtS	60	2 hours ago	delete
test_user1(block)	Fake Input	QfehQpttam	54	2 hours ago	delete
test_user1(block)	Fake Input	eyOCMyQJnI	11	2 hours ago	delete
		<- Show previous	Show:	Show next ->

The table will display a bunch of data. Which is nice but additionally I added sorting, changing the number of translations per page and blocking users. The sorting can be done by user, language, upvotes, post time and in ascending and descending order.

The code will basically just create the query for different parameters. I am a firm believer of avoiding code repetition, not because of the whole if it is in one place it is easier to update philosophy (I am not saying this is not a big benefit) but more because I want to believe that the code can indeed be 'smart'.

Why the blocked users?

I had a dream about a world where people would not spam, troll and inject - I am sure someone said it, but if not I just did.

The point is that although we would like to think that the plugin users will submit only what is expected that is not the case, that is why we (in general) do input validation, security checks and ... blocking users. The blocked user will be prevented from submitting translations, voting, all of his translations (and their votes) will be deleted and he will be stripped from his badges. If you get blocked this is how the plugin looks like:

The last part of the admin interface allows to show mercy upon the poor soul:

while (problem) { me.ask_for_help(); }

The My Badges section displays all the badges the user has acquired over time. We wont to use
Gamification here to encourage users to submit translations and vote translations.

The preliminary list of 'badges' a user can get is:

• First translation
• 10th, 50th, 100th... translation
• First vote
• Casting a vote which removes a translation
• Getting more than 100 upvotes
• Submitting more than 10 translations in a day
• Submitting more than 20 translations in a day
• Submitting translations 1, 7, 20 days in a row
• Working on a language which is fully translated

This list will be a bit longer, If you have an idea for awards and designs please do send me an email or leave a comment.

Non related stuff

Sarajevo had it's second startup weekend this weekend. As this is a great opportunity to meet people with the same interests I was in line for it. Now someone might (and did say) : "You are crazy, you spend your weekend working and pay for it" - +Lasse Chor . Thinking about it , does seem a bit crazy. But even whit the back-pain (bending over a laptop for 2 days and I do mean bending), after having a couple of  0-100

 range bursts it was great and I can't wait to do it again. Not only did I get to meet some great people, even a real 'Googler' +Martin Omander , he was the man that made my team work together.
And I got to discover that I am not even that bad at presenting. Well I was but with quite a bit of a kick-start from +Nejra Plasto I got it somewhat down.

All in all a great weekend, and for the second time in my life I got to be called a marketing genius, and taking from whom I got this compliment I am proud.

But, I was and will be a programmer/developer first and then we can talk about all the other stuff.

The plugins index page

Before I start with the blog I have to share this picture which is on my desktop right now just as a silent reminder:

So the time came to go ahead an give something to the user. As mentioned here Geeklog has a specific structure for plugins. What I did here is the public_html/plugin_name/index.php or simply the page displayed after the user clicks on the navigation link of the plugin.

The hardest part here was deciding on what to display, I decided to go with some stats and some basic functionality. And of course displaying the plugin info.
I also started implementing lib-translator.php which will hold all the function I use for the plugin. Including AJAX call function. In lack of better idea the AJAX calls will pass one additional parameter, the name of the function they need, the library will then call the function. Something like this:

Lets see how it all looks

For a anonymous user (not logged in) the plugin page is very,very simple it displays only the info about the plugin including that it is done as part of +Google Summer of Code  by me under the mentorship of +Dirk Haun .

After the user simply logs in (as in has not jet used the plugin) the page will not change much. It will show a preview of what it will have once the hands are dirty from translating.

The Plugin info is hidden, it has the same content as if the user was not logged in and the view can be toggled by hitting the show link. The show  will simply display the content by removing the hidden  CSS class I use. As for the badges the process is different, it will initially display only 4 of them, show all  will issue a AJAX call and get the rest of them (The page will not reload).

Now after a a few translations have been submitted, the page gets a bit more vivid.

The Quick Stats part displays a few number for the user and general numbers - Number of translations by user, Number of translations in total... It also displays a list of languages the user has worked on and their current translation percentage, as in the plugin side form.

The My Badges  section displays all the badges the user has acquired over time. We wont to use
Gamification here to encourage users to submit translations and vote translations.

The preliminary list of 'badges' a user can get is:

• First translation
• 10th, 50th, 100th... translation
• First vote
• Casting a vote which removes a translation
• Getting more than 100 upvotes
• Submitting more than 10 translations in a day
• Submitting more than 20 translations in a day
• Submitting translations 1, 7, 20 days in a row
• Working on a language which is fully translated

This list will be a bit longer, If you have an idea for awards and designs please do send me an email or leave a comment.

This is how a single badge should look like:

The On a Run  is the title of the badge, and the tooltip is there to explain for what a badge is given.

Note The images used here are for testing purposes only!

The last thing left on the page is the list of translations:

The list will display all translations submitted by the user in increasing order of time passed since submission. And allow the user to delete a translation. It does not allow for changes as I think it is much better to do translating in context, and anyhow after changing a translation all the data would be rested which would make it equal to submitting a new translation. The list will have highlighted translations with a negative number of votes.

The idea is to delete translations after they reach a certain number of negative votes. The number will be set in the plugin configuration by the admin.

Not related stuff

As the summer goes by and it gets hot in my room I figured work would be so much easier with a bit of music, so here it is my working playlist.