Workaround for conversion of Unicode Vietnamese to ANSI

A few months ago, we had an interesting project which involved Vietnamese. As a part of the project, we have hit a minor snag.

It looks like the first support for Vietnamese appeared in 1990s, which means the ANSI codepage was built in a hurry already when Unicode was either in the works or already out. The tonal nature of the language demands complex combinations of diacritics to be used. Not just regular grave accents, umlauts and such. Simply put, there were not enough slots for the new characters, so the creators of the Vietnamese codepage stuffed these new characters wherever, in whatever way possible.

Today very few people use ANSI, however, it is still needed for several reasons: legacy being one (people still work with mainframes, you know), and compliance another. Of course, there is a tried and true function WideStringToMultiByte which works like a Swiss chronometre. That is, for most languages - except Vietnamese. There are posts by Microsoft folks stating that "Vietnamese is a complex language on Windows" (duh!), but not really telling how to fix it. I asked around, no one replied, as expected (I love how Stackoverflow people react when they can't answer the question :-) ).

After scrutinising the result I saw what's the problem. It seems that the decomposing routine during the conversion is unable to handle some combinations of Unicode characters. Manually decomposing some characters to their equivalents worked for me.

I use an esoteric language called Clarion to design our tools and some components, so my original code is in Clarion. A few days ago, Mark Jacobs from Critical Research contacted me requesting help with the same issue, and kindly converted my Clarion source code to C++ more familiar to the rest of the world. Thanks, Mark!

Get Clarion source code here and Mark's C++ here.

LinguaSys - USA Today

An article about an Android app built by our partners in LinguaSys using Carabao as a backend (among others), is published in USA Today and other news sources.

ALTA 2010

Digital Sonata was invited to present at the Australian Language Technology Association 2010 workshop on Thursday, December 9, 2010. Overview and directions are here.

Vadim Berman will be speaking some time between 3:30pm and 5:30pm.

Digital Sonata Signs Long Term Exclusive Agreement with LinguaSys™ For Use Of Carabao For Machine Translation

Digital Sonata signed a long term deal with LinguaSys™ for the exclusive use of Carabao in machine translation (MT) solutions on March 24, 2010.

Carabao is a hybrid language translation system using both statistical and rules-based methodologies. Vadim Berman, CEO of Digital Sonata in Australia, and Chief Technology Officer and a co-founder of LinguaSys, is the author of Carabao. Berman has a wealth of dedicated experience in the field of MT and text analysis.

Brian Garr, CEO of LinguaSys said, “We are very excited that this incredible technology from Digital Sonata will help us create the next generation of language translation solutions.”

LinguaSys is a new next generation machine translation company. LinguaSys’ Carabao language middleware uses language processing methodologies offering excellent comprehension in the least amount of time at low cost. LinguaSys enables enterprises to translate volumes of information, including text chat, e-mail, web pages and documents, quickly, accurately and automatically. LinguaSys provides the creation of new MT languages, customized lexical services, ease of use, compatibility with existing natural language software, security behind the firewall, availability, integration and lower memory requirements.

Carabao Language Kit 1.7.0.0 released

The version 1.7.0.0 is now available for download.

Fixed:

• Handling of control priority greater than 2, when some of the members have no feasible agreement graph. The result was, that some parts of the sequence worked, and some didn't.
• Truncation of very long sentences

Added:

• A utility to validate and correct rule unit values
  
• A generic support for formatted processing, e.g. HTML, XML, SGML including embedded formatting elements in the text flow
• GUI to test formatted processing in Carabao Test Console
• Automatic conversion of double-byte space characters into standard single-byte

Improved:

• Regular expressions for segmentation into character classes for double-byte languages
• Perl-compatible regular expressions have been introduced for unknown heuristics
• Frequency-based backtracking added to the tokenization algorithm
• Unicode clipboard support in Carabao desktop suites is now bidirectional: when leaving the application and when coming back to the application

Publication in Multilingual

My article on evaluation of emerging language technologies was published in Multingual:

http://multilingual.texterity.com/multilingual/20100405?pg=59#pg64

English - Swedish OLIF dictionary released

Engish - Swedish OLIF dictionary added to the list of OLIF lexicons distributed by Digital Sonata. The dictionary is available for download from http://www.digitalsonata.com/download.aspx?type=linguisticData.

Bilingual OLIF dictionaries released

Digital Sonata released a set of low-cost royalty-free bilingual dictionaries in OLIF format, optimized for use in NLP and content management applications. Translation, part of the speech, and a thesaurus article is included. The dictionaries are available at http://www.digitalsonata.com/download.aspx?type=linguisticData. Currently the following dictionaries are available:

• English -> Finnish


• English -> French


• English -> German


• English -> Japanese


• English -> Korean


• English -> Russian


• English -> Spanish

Carabao Language Kit 1.6.2.1 released

The version 1.6.2.1 is now available for download.

Fixed:

• Transliteration to empty string
• Partial transliteration

Added:

• Change log which allows distributed collaboration

Improved:

• Processing speed
• Entry matching accuracy

Running IIS7 on Windows Vista Home

Many people are stumbled when trying to use their Windows Vista Home machines for web development involving IIS deployment. Usually the tricky part is the error message that ASPX pages are not handled. As the management console is not installed on some Vista editions, the workaround is to edit the configuration file directly.

The configuration file is C:\Windows\System32\inetsrv\config\applicationHost.config. Simply overwrite the <handlers> section with the following:


<handlers accessPolicy="Script, Read">
<add name="TraceHandler-Integrated" path="trace.axd" verb="GET,HEAD,POST,DEBUG" type="System.Web.Handlers.TraceHandler" preCondition="integratedMode" />
<add name="WebAdminHandler-Integrated" path="WebAdmin.axd" verb="GET,DEBUG" type="System.Web.Handlers.WebAdminHandler" preCondition="integratedMode" />
<add name="AssemblyResourceLoader-Integrated" path="WebResource.axd" verb="GET,DEBUG" type="System.Web.Handlers.AssemblyResourceLoader" preCondition="integratedMode" />
<add name="PageHandlerFactory-Integrated" path="*.aspx" verb="GET,HEAD,POST,DEBUG" type="System.Web.UI.PageHandlerFactory" preCondition="integratedMode" />
<add name="SimpleHandlerFactory-Integrated" path="*.ashx" verb="GET,HEAD,POST,DEBUG" type="System.Web.UI.SimpleHandlerFactory" preCondition="integratedMode" />
<add name="WebServiceHandlerFactory-Integrated" path="*.asmx" verb="GET,HEAD,POST,DEBUG" type="System.Web.Services.Protocols.WebServiceHandlerFactory, System.Web.Services, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" preCondition="integratedMode" />
<add name="HttpRemotingHandlerFactory-rem-Integrated" path="*.rem" verb="GET,HEAD,POST,DEBUG" type="System.Runtime.Remoting.Channels.Http.HttpRemotingHandlerFactory, System.Runtime.Remoting, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" preCondition="integratedMode" />
<add name="HttpRemotingHandlerFactory-soap-Integrated" path="*.soap" verb="GET,HEAD,POST,DEBUG" type="System.Runtime.Remoting.Channels.Http.HttpRemotingHandlerFactory, System.Runtime.Remoting, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" preCondition="integratedMode" />
<add name="AXD-ISAPI-2.0" path="*.axd" verb="GET,HEAD,POST,DEBUG" modules="IsapiModule" scriptProcessor="%windir%\Microsoft.NET\Framework\v2.0.50727\aspnet_isapi.dll" preCondition="classicMode,runtimeVersionv2.0,bitness32" responseBufferLimit="0" />
<add name="PageHandlerFactory-ISAPI-2.0" path="*.aspx" verb="GET,HEAD,POST,DEBUG" modules="IsapiModule" scriptProcessor="%windir%\Microsoft.NET\Framework\v2.0.50727\aspnet_isapi.dll" preCondition="classicMode,runtimeVersionv2.0,bitness32" responseBufferLimit="0" />
<add name="SimpleHandlerFactory-ISAPI-2.0" path="*.ashx" verb="GET,HEAD,POST,DEBUG" modules="IsapiModule" scriptProcessor="%windir%\Microsoft.NET\Framework\v2.0.50727\aspnet_isapi.dll" preCondition="classicMode,runtimeVersionv2.0,bitness32" responseBufferLimit="0" />
<add name="WebServiceHandlerFactory-ISAPI-2.0" path="*.asmx" verb="GET,HEAD,POST,DEBUG" modules="IsapiModule" scriptProcessor="%windir%\Microsoft.NET\Framework\v2.0.50727\aspnet_isapi.dll" preCondition="classicMode,runtimeVersionv2.0,bitness32" responseBufferLimit="0" />
<add name="HttpRemotingHandlerFactory-rem-ISAPI-2.0" path="*.rem" verb="GET,HEAD,POST,DEBUG" modules="IsapiModule" scriptProcessor="%windir%\Microsoft.NET\Framework\v2.0.50727\aspnet_isapi.dll" preCondition="classicMode,runtimeVersionv2.0,bitness32" responseBufferLimit="0" />
<add name="HttpRemotingHandlerFactory-soap-ISAPI-2.0" path="*.soap" verb="GET,HEAD,POST,DEBUG" modules="IsapiModule" scriptProcessor="%windir%\Microsoft.NET\Framework\v2.0.50727\aspnet_isapi.dll" preCondition="classicMode,runtimeVersionv2.0,bitness32" responseBufferLimit="0" />
<add name="ASPClassic" path="*.asp" verb="GET,HEAD,POST" modules="IsapiModule" scriptProcessor="%windir%\system32\inetsrv\asp.dll" resourceType="File" />
<add name="SecurityCertificate" path="*.cer" verb="GET,HEAD,POST" modules="IsapiModule" scriptProcessor="%windir%\system32\inetsrv\asp.dll" resourceType="File" />
<add name="SSINC-stm" path="*.stm" verb="GET,POST" modules="ServerSideIncludeModule" resourceType="File" />
<add name="SSINC-shtm" path="*.shtm" verb="GET,POST" modules="ServerSideIncludeModule" resourceType="File" />
<add name="SSINC-shtml" path="*.shtml" verb="GET,POST" modules="ServerSideIncludeModule" resourceType="File" />
<add name="ISAPI-dll" path="*.dll" verb="*" modules="IsapiModule" resourceType="File" requireAccess="Execute" allowPathInfo="true" />
<add name="CGI-exe" path="*.exe" verb="*" modules="CgiModule" resourceType="File" requireAccess="Execute" allowPathInfo="true" />

</handlers>

Done.

BlackBerry connection tip

We developed a small client application for BlackBerry phones that connects to MiaMia servers. A lot of things can be said about BlackBerry adherence to standards, but I don't want to use cuss words here. The main problem is that these phones were initially devised to serve as "corporate walkie-talkies", and only recently their design was stretched to accommodate the needs of the general public. In a nutshell, this is a hardware equivalent of Lotus Notes - a combination of lack of respect for standards and a kungfu grip on the corporate customers.

As a result, once in a while developers encounter what can be described as a "chastity belt" - when it comes to communications with external world. Look for blackberry connection problems in Google or read this frustrated rant that sums it all. And if, as a BlackBerry user, you encountered problems with MSN Messenger, IRC, or anything else - same reason.

By default, when possible and a proxy-based connection (MDS, BES, BIS) exists, BlackBerry seems to re-route all requests there. It is not at all certain that your tiny app will be allowed to go further. Chances are, it will be stopped, and possibly hang. However, there are two possibilities that should work in most cases:

• direct TCP/IP via GPRS network
• wireless connection

To cut the long story short, the easiest way to provide maximum coverage is to create a tiny function that will be used like this:

 protected String getDeviceSpecificUrlParameters()
{
      try {
 if (net.rim.device.api.system.WLANInfo.getWLANState() == net.rim.device.api.system.WLANInfo.WLAN_STATE_CONNECTED)
              {
  return ";deviceside=true;interface=wifi;ConnectionSetup=delayed;retrynocontext=true";
              }
          } catch (Exception e) {} // doesn't matter, just go elsewhere
 return ";deviceside=true";
}

then call it whenever you make a connection, e.g.

 ...
Connector.open(url + getDeviceSpecificUrlParameters());
...

If your application is designed to work across different platforms (like ours), you can re-implement that getDeviceSpecificUrlParameters() for whatever platform you need.

A word of warning: if it's a brand new phone, and the access to GPRS is not configured, you may need to explain your users how to set up their APN parameters. The carrier-specific parameters can be found here, here, and here.

Making peace with Vista

This is not directly related to the subject of the blog, but is worth sharing, since so many people are having problems with this.

My development machine runs Windows Vista. I'm not a fan of it (probably, like the majority), but I have to eat the same food my users do. Many people experience the same phenomenon, where Vista suddenly freezes without any provocation. It gets progressively worse with time. In my case, a few days ago it came to a point where the freezing occurred every half an hour. 

I tried different things: switching off visual styles, removing the useless Windows Defender, etc. The performance improved, of course, but the freeze attacks still remained.

Finally, I found this tip. (In Vista Home Basic, it is a bit different: you need to navigate to Control Panel -> System -> Performance -> Adjust indexing options.) It all makes sense: usually in databases functional indices are paramount, and when the system builds them, everything else is secondary. The most important analogy: if an index is corrupt, all kinds of bad things are likely to happen. Except that why would we need them in a desktop OS? It turns out that the reason is the ultra-annoying search, which is a part of the Start menu. I suppose the reason for this nuisance was to show that it's no worse OS X. Well, this is dumb. The old search was fine, and those who need better capabilities, generally use something else. In Vista, as a result, a background process is indexing nearly every file in the system. Why not make the indexing a scheduled task, and fall back to un-indexed search when necessary?

After I reduced the population of files to index to the contents of Start menu only, suddenly Vista became the most stable, high-performance system I have ever seen - I kid you not! On the other hand, if you do need and like that new bloated search, or need the thumbnails, simply go and rebuild the index. Also at this opportunity review what extensions you want to index.

Oh, and of course, I assume no responsibility if you do something silly to your machine, so don't just push everything without thinking.

Guide to sequences uploaded

We uploaded a short guide to building and debugging the sequences. It is available at our whitepaper download page.

Carabao Language Kit 1.5.0.1 released

The version 1.5.0.1 is now available for download. Lots of changes and enhancements thanks to ongoing development of Chinese (not in the default database though).

Fixed:

• Regression: "phantom capitalization" of re-used words
• Regression: sequence style forcing / avoiding
  
• Repositioning errors in sentences with attached tokens
• Sequence processing in languages not using white spaces
• Regression: single member sequence processing

Added:

• Lattice-based processing for speech recognition and OCR application usage
• Optional and unmapped members in sequences
• Members in sequences which are validated but not mapped
• Possibility to get a crosslingual representation (components only: DeepAnalyzer and Translation Server)
• Possibility to load content from a disambiguated crosslingual representation
• GUI in Translation Console to enable lattice-based processing
• GUI in Translation Console to enable loading crosslingual representation
• GUI in Translation Console to hint the system about the expected domains in the text
• Analysis mode in Translation Console, when the source and target languages are the same and no styles are enforced / avoided
• Capability of using the white space as a delimiter in languages that don't have white spaces
• Smart quotes and other delimiters

Improved:

• Dictionary GUI - presents thesaurus from another language, if missing in the current language
• Sequence builder GUI - color coding of members which are not mapped, or contain conditions producing empty sets

Carabao Language Kit 1.2.3.0 released

The version 1.2.3.0 is now available for download.

Fixed:

• Handling of single quotes as syntax delimiters in English

Added:

• A segmentation mode more effectively handling languages that don't use white spaces (e.g. Chinese, Japanese, Korean, Thai). In this mode, different character classes are broken into tokens (e.g. Chinese, and then immediately English). The remaining unidentified part is run through unknown heuristic identifier.


• Automatic conversion for Unicode clipboard data into the currently active encoding in tokens table


• Better warning when attempting to overwrite the current token


• A utility to rebuild semantic links cache

Improved:

• In some systems, the table of tokens with every update was adding a new set of system icons (minimize, restore, maximize) to the MDI frame window. The maximize option now causes the window to be set roughly to the full client area, but not in maximize mode

Free source code section

We added a small source code section on our Download page, where we will post freebies for developers.

Carabao Language Kit 1.2.0.0 released

The version 1.2.0.0 is now available for download.

Fixed:

• Unknown patterns were translated as hypernyms


• Regression: certain category-based sequences were omitted on second execution because of a malfunctioning guess scan caching mechanism


• In analytical mode (Carabao DeepAnalyzer), there was a mismatch between word index number and an idiom member index, in sentences with attached tokens such as 'em, 'm


• When copying a token with 1 rule units or less, the text is always reset to the original

Added:

• Capability to match numbers as patterns


• When a translation is not found, the engine tries to fall back to a matching hypernym instead


• New methods to Carabao DeepAnalyzer that enable accessing the members of the detected idioms


• New methods to Carabao CDA that enable accessing the unknown heuristics table


• New sequences


• Russian morphological exceptions

Improved:

• If an "unknown pattern" is forced to match a known word, it will not create a new guess if a guess with a same hypernym already exists. For example, if you force to check, whether a known word can be a city, a new record will not be created, if there is already a guess with a known city
  Automatic input language switching in locator fields
  
• Locator fields are pre-filled with the list of all existing languages in the database, eliminating the need to jump to the next language

The surreal world of VoIP

We're currently working on a massive project involving telephony and mobile technologies, and I had to look for VoIP vendors to cater for relatively simple needs of my client. I have to say that while the needs are simple, the traffic is extremely high, so in monetary terms, it could be a nice deal for the VoIP vendors.

But... is VoIP a strange industry or what. I don't know who runs all these companies - but:

• if you don't know what IVR, DID, PSTN, or all their other "secret handshakes" mean - they won't even talk to you. Forget about the forums, they are even less helpful than Usenet.
  
• the responses come usually after weeks, and they are of the type "I just transferred your inquiry to our sales representative". Obviously, unless you kick and scream, the sales rep won't get back
• you have to re-tell the story over and over again

Ah yes, but the internet is full of dotcomish optimism about mashups and other kewl stuff. Awesome, dude.

Of course, there are also people who do need paying customers, and those who are able to concentrate, and - surprise surprise - they were the ones who got the job eventually.

Published in MultiLingual

My article about real-world applications of machine translation has been published in MultiLingual Computing, the leading industry magazine for globalization, international software development and language technology.

This part is for subscribers only though.

Localization can be a matter of life and death

This story (A Cellphone's Missing Dot Kills Two People, Puts Three More in Jail) reads like it was taken from a Tarantino or Coen brothers' movie.