vvIfs_readFile() and CCSIDs

[rkanemeier]

I created a CSV file from a PF using CPYTOIMPF. Here was my command:

CPYTOIMPF FROMFILE(FPVAL41DEV/MYTFILE) TOSTMF('/a-file-import.csv') STMFCCSID(*PCASCII) RCDDLM(*CRLF) RMVBLANK(*BOTH)

I then created an RPG program using the vvIfs_readfile() example.

fileHandle = vvIfs_openFile('/a-file-import.csv'':OREADONLY);
                                                           
x=1;                                                       
dou status = -1 or x = %elem(record);                      
                                                           
  data = vvIfs_readFile(fileHandle:status:READLINE);       
  data = vvUtility_decodeUTF16(data);                      
                                                           
  record(x) = data;                                        
  x+=1;                                                    
enddo;                                                     
                                                           
vvIfs_closeFile(fileHandle);

When I inspect the data it looks encoding (lots of umlauts, vowel expressions etc). How do I c0nvert the data in the "proper" format to update a PF? (is this a CCSID issue)?

[robert.swanson]

It's may be a CCSID issue, but it could also be that the data is being mangled when you do the vvUtility_decodeUTF16() call -- that's usually only needed for encoded hex data coming from the browser, which I don't think applies here.

So I would suggest you remove that line first and see what happens. If it's still not looking right, try experimenting with adding a FROMCCSID value on your CPYTOIMPF command, changing the STMFCCSID value, and/or specifying a CCSID as the third parm on your vvIFS_openFile() call.

I would also suggest you adjust your loop to enter with x=0, then after the vvIFS_readFile you should test to see if status is not -1, then add 1 to x and set record(x)=data. As it's currently written you'll be setting record(x) even when status is -1.

[rkanemeier]

Followed your coding recommendations (sans openFile) and still having the same issue.

Changed the CCSID of the IFS file from 1252 to 819 (used CHGATR). Still having the same issue.

I am confused about your recommendation of specifying the CCSID as the 3rd parameter of the openFile procedure. The Valence API documentation shows that the 3rd and 4th parameters are used for creating an IFS file, and the CCSID option is the 4th parameter. For this application, we are using Valence 4.1. Is this possibly the confusion?

What CCSID does the readFile procedure expect?

Here's my new code:

fileHandle = vvIfs_openFile('/a-file-import.csv':OREADONLY);    
                                                               
x=0;                                                           
dou status = -1 or x = %elem(record);                          
                                                               
  data = vvIfs_readFile(fileHandle:status:READLINE);           
                                                               
  x+=1;                                                        
  record(x) = data;                                            
enddo;                                                         
                                                               
vvIfs_closeFile(fileHandle);

[robert.swanson]

Sorry, I meant to say the CCSID is the 4th parm on vvIfs_openFile(). You can see an example of it being applied in the vvIfs_openFile() procedure in the RPG source for VVADM_APPS, in the getSettings procedure.

Whenever you're dealing with IFS logic, I find it's best to write a simple standalone program to prove things out in debug mode, then copy the working logic into your Valence program once you know the data is being retrieved properly.

Keep in mind, the VVIFS open and read procedures are just simple wrappers to the equivalent IBM APIs, which are more thoroughly documented here and here. You can also glean some useful tips from Scott Klement's very well-written piece, Working with the IFS in RPG.

I would also add that your latest code, as written, will still be adding an extra element to your record array even when there's no data retrieved from readFile.

[rkanemeier]

Thanks Rob.

My code above is a standalone program that I'm using to vet my theory that reading a csv file is better than creating a PF in QTEMP with the same record layout and using CPYFRMIMPF to get the data from the csv into a PF. However, this is turning out to be more problematic than I anticipated. I remember reading Scott Klement's IFS tools guide and had worked through several examples. I remember several forum posts that state you have to create the IFS file first, then put data into it, then read it, or you get unexpected results (maybe my memory of this is false). That being said, it is kind of strange that the CCSID of the files being uploaded is 1252, even though the in_file documentation says it defaults to 819 if not specified. In either case, I've already tried 819 and it doesn't work either.

My code is just test code to prove it the concept. I it's not fool proof. ;-)

Thanks for your help.


As a side note, I found this comment by Scott:

Finally... when you specify a CCSID in the 4th parameter, it's intended to tell the API the CCSID of the variables in your program -- NOT the CCSID of the file. (It already knows the CCSID of the file, since the file has been marked with a CCSID in the IFS) So by specifying 819, you've effectively told it to translate the file *to* 819... In other words, if the file is in EBCDIC, it'll translate it to ASCII in your program. (If the file is already ASCII, it'd do no translation)

So it seems that 4th parameter is used to translate from the IFS file's CCSID to a CCSID your program variables are in, then 37 would be the appropriate value. However, that did not work either. ;-)

[robert.swanson]

What you're trying to do should certainly be workable with these APIs, just a matter of whittling it down to find which setting is the culprit. Might be worth creating a csv file on the IFS from RPG (i.e., via vvOut_execSQLtoCSV(), with vvOut.file set to 'F') and verifying your logic can read it, then compare that file's properties in WRKLNK with the csv file you're uploading.

If it's any consolation, anytime we've had to write logic to pull data from the IFS it almost never works right on the first or second attempt.