Announcement

Collapse
No announcement yet.

Need Help to address passing Non-Ascii characters as parameter

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Need Help to address passing Non-Ascii characters as parameter

    I tried to pass UTF-8 character set through body in Valence API call, but the RPG program losts all non-ASCII characters. How can I address passing UTF-8 character set data or any non-ascii data to rpg program?

  • #2
    For standard character fields, there is a conversion that takes place between UTF-8 (CCSID 1208) and the IBM i job on the server, which is typically English CCSID 37. English CCSID 37 uses an EBCDIC encoding, not ASCII. For CCSID 37 you would be limited to the characters in that code set. See doc here: https://en.wikipedia.org/wiki/EBCDIC_037. RPG programs are certainly capable of handling UTF8 (more typically UTF16), but you first need to encode the data on UI side, pass it to the RPG, then decode it. Look at vvIn_utf16 to get some idea: http://service.cnxcorp.com/valence-5...G_Toolkit.vvIn.

    Comment


    • #3
      My valence job is set to run with Job's CCSID 65535 to correctly read UTF-8 characters. When I am passing UTF-8 character set in body as JSON data, VVIN program gets the jobs CCSID ad 37 in procedure vvin_body ("currentCCSID = vvUtility_getCurrentCCSID();"). Subsequently it converts UTF-8 data to CCSID 37. This causes my input data to (which contains Greek data) is converted into junk. To my understanding the following two items are not happening correctly:
      1. Jobs CCSID should have been returned as 65535.
      2. If the input character set is UTF-8, no conversion should take place.

      Comment


      • #4
        I would be surprised if your Valence CGI jobs are running with CCSID 65535. Your system value QCCSID is probably set to 65535 but the Apache CGI jobs will always start up with a non-65535 CCSID. Your CCSID of the Apache jobs would be 37 for an English system. Look at jobs under the QHTTPSVR subsystem, put a 5, then 2, then scroll down to see the CCSID of the job that is handling your call. You will not want these jobs to run in 65535 because they need to do conversions for all of the data being passed outside of your UTF-8 data.

        Try this: Go into Portal Admin-->Users. For any user's description put in some Japanese or other characters you're having trouble with. Save that and make sure you can save and reopen that user record an have it return the description just as you entered. Now open up Google Chrome dev tools and look how the data is being passed. It is encode on the UI, then decoded on the backend, and vice-versa on the return trip.

        If you still can't get it to work, just post up some sample data to this thread that I can test with and I will post you back a program example using vvIn_body.

        Comment


        • #5
          I tried to pass the following data as JSON object with encoding UTF-8 from body of the API.
          {
          "HEADER":
          {
          "TITLE": "MR",
          "FIRSTNAME": "????r?",
          "LASTNAME": "???????????",
          "ADDRESSLINE1": "KATSIMPIRI 32",
          "ADDRESSLINE2": "",
          "ADDRESSLINE3": "",
          "ADDRESSLINE4": "",
          "CITY": "HOLARGOS",
          "STATE": "GRC",
          "COUNTRY": "GRC",
          "POSTALCODE": "155 61",
          "PHONENUMBER": "4803368615",
          "EMAILADDRESS": "TestGuestljoin1A1a@usa.la",
          "APPLICANTHOMECOMPANY": "GRC",
          "APPLICANTCREATEDDATE":"20180620",
          "SPONSORID": "001000000001",
          "SPONSORHOME": "GBR",
          "MEMBERTYPE": "Guest",
          "PRIVACYPREFERENCE": "",
          "EMAILFLAG": "",
          "LANGUAGEPREFERENCECODE": "",
          "SPONSORWIZARD": "AUTO",
          "ALLOWSPONSORCONTACTBYEMAIL": "Y",
          "ALLOWSPONSORCONTACTBYPHONE": "Y"
          },
          "AUXDATA": [

          ],
          "AUXTOTALCOUNT": 0
          }

          The special characters in First Name and Last Name are not displayed correctly, in Edit mode it appears correctly. Can you please test for the First Name and Last Name are captured correctly through vvin_body.
          Last edited by abhakat_FLP1; Yesterday, 02:14 PM.

          Comment


          • #6
            The first part of this is to encode your fields prior to sending them to the RPG. Standard way to do this with is using Valence.util.Helper.encodeUTF16. For example, using Valence.util.Helper.encodeUTF16("????r?") on FIRSTNAME becomes "0393039403A003A3007203BE" and LASTNAME becomes "039A03BF03C503C403C303CC03C003BF03C503BB0399" .

            Valence.util.Helper.encodeUTF16 is available to all Valence apps when valence.js is pulled in as part of the index.html. If you need to use the encodeUTF16 logic separately from a Valence UI here is the JavaScript function you can use directly:

            Code:
                  function ecodeUTF16(textString) {
                        var haut = 0;
                        var n = 0;
                        var CPstring = '';
                        var dec2hex = function(textString) {
                                return (textString + 0).toString(16).toUpperCase();
                            };
                        try {
                            for (var i = 0; i < textString.length; i++) {
                                var b = textString.charCodeAt(i);
                                if (b < 0 || b > 65535) {
                                    CPstring += 'Error: byte out of range ' + dec2hex(b) + '!';
                                }
                                if (haut !== 0) {
                                    if (56320 <= b && b <= 57343) {
                                        CPstring += dec2hex(65536 + ((haut - 55296) << 10) + (b - 56320));
                                        haut = 0;
            
                                        continue;
                                    } else {
                                        CPstring += 'Error in: surrogate out of range ' + dec2hex(haut) + '!';
                                        haut = 0;
                                    }
                                }
                                if (55296 <= b && b <= 56319) {
                                    haut = b;
                                } else {
                                    cp = dec2hex(b);
                                    while (cp.length < 4) {
                                        cp = '0' + cp;
                                    }
                                    CPstring += cp;
                                }
                            }
                        } catch (e) {}
                        return CPstring;
                    }


            So your sample data would look like this once the relevant fields are encoded:

            Code:
            {
                "HEADER": {
                    "TITLE": "MR",
                    "FIRSTNAME": "0393039403A003A3007203BE",
                    "LASTNAME": "039A03BF03C503C403C303CC03C003BF03C503BB0399",
                    "ADDRESSLINE1": "KATSIMPIRI 32",
                    "ADDRESSLINE2": "",
                    "ADDRESSLINE3": "",
                    "ADDRESSLINE4": "",
                    "CITY": "HOLARGOS",
                    "STATE": "GRC",
                    "COUNTRY": "GRC",
                    "POSTALCODE": "155 61",
                    "PHONENUMBER": "4803368615",
                    "EMAILADDRESS": "TestGuestljoin1A1a@usa.la",
                    "APPLICANTHOMECOMPANY": "GRC",
                    "APPLICANTCREATEDDATE": "20180620",
                    "SPONSORID": "001000000001",
                    "SPONSORHOME": "GBR",
                    "MEMBERTYPE": "Guest",
                    "PRIVACYPREFERENCE": "",
                    "EMAILFLAG": "",
                    "LANGUAGEPREFERENCECODE": "",
                    "SPONSORWIZARD": "AUTO",
                    "ALLOWSPONSORCONTACTBYEMAIL": "Y",
                    "ALLOWSPONSORCONTACTBYPHONE": "Y"
                },
                "AUXDATA": [],
                "AUXTOTALCOUNT": 0
            }
            In the next post I will show you how to pull in and decode the variables in RPG.

            Comment


            • #7
              Here is the RPG program that shows how to pull in the FIRSTNAME field. Your goal is to get the field into a "C" type field which is ucs2/utf16, a typical field used to hold unicode data within RPG. First you bring in the encoded data into a standard A field (firstNameA), then use vvUtility_decodeUTF16 to convert it to the ucs2 field. The ucs2 field can then typically be written to the database. Be sure you are writing to a database field that is compatible with unicode. We strongly recommend CCSID 1200 (UTF16). You can not hold this data in standard character field unless you want to store the data in the encoded format and decode it when you pull it out.

              Code:
                    /copy qcpylesrc,vvHspec                                                             
                   ** --------------------------------------------------------------                    
                   **         Copyright (C) 2008-2018 CNX Corporation                                   
                   ** --------------------------------------------------------------                    
                   **     Object ID: EXUTF16                                                            
                   **       Version: V5.2                                                               
                   **   Description: Example of pulling in UTF16 data from vvIn_body                    
                   ** --------------------------------------------------------------                    
                    /define includeJSON                                                                 
                    /include qcpylesrc,vvDspec                                                          
                   **--------------------------------------------                                       
                   d body            s          65535a                                                  
                   d pointer1        s               *                                                  
                   d pointer2        s               *                                                  
                   d firstNameA      s            120a                                        encoded   
                   d firstNameC      s            120c                                        unencoded 
                   **--------------------------------------------                                       
                   **  program start                                                                    
                   **--------------------------------------------                                       
                    /free                                                                               
              
                     // get the body                                                                    
                     body=vvIn_body();                                                                  
              
                     // parse the json                                                                  
                     pointer1=vvJson_parse(%addr(body));                                                
              
                     // get the HEADER object from json                                                 
                     pointer2=vvJson_getObject(pointer1:'HEADER');                                      
              
                     // get utf-16 encoded variable                                                     
                     firstNameA=%str(vvJson_getString(pointer2:'FIRSTNAME'));                           
              
                     // decode the utf-16 variable into a ucs2 field                                    
                     firstNameC=vvUtility_decodeUTF16(firstNameA);                                      
              
                     // send sucess response                                                            
                     vvOut_toJsonPair('success:true');                                                  
              
                    *inlr=*on;                                                                         
                   /end-free

              Comment

              Working...
              X