***** Creating CSV files from glycoCT files *****

CSV files were created by converting glycoCT files to csv files using the java program called GTC2CSV, which has the following command-line syntax:
Usage: java -jar GCT2csv.jar input_file [list | single (default)] [0 <= verbosity <= 5 (default = 0)]


- The input file can be either a list of files or a single file: if the second argument is "list", then the input file is processed as a list, otherwise it is processed as an individual input file.  Entries in the list file should  have  fully specified paths (see example below).
- the output (csv) files are written to a subdirectory (csv) of the directory containing each glycan structure file that is processed

Here is an example using a single input glycoCt file.

$java -jar GCT2csv.jar base/GlycoCT/G00176HZ.txt single 9

NOTE! - the "$" is the command prompt on my machine - This is NOT part of the command
NOTE! - "base" is a symbolic link to a directory containing the directory that itself contains the GlycoCT files - I think showing all of the details of my computer's directory system would just be confusing.

### base/GlycoCT/G00176HZ.txt ###
RES
1b:b-dglc-HEX-1:5
2s:n-acetyl
3b:b-dglc-HEX-1:5
4s:n-acetyl
5b:b-dman-HEX-1:5
6b:a-dman-HEX-1:5
7b:b-dglc-HEX-1:5
8s:n-acetyl
9b:b-dglc-HEX-1:5
10s:n-acetyl
11b:a-dman-HEX-1:5
12b:b-dglc-HEX-1:5
13s:n-acetyl
14b:b-dglc-HEX-1:5
15s:n-acetyl
16b:a-lgal-HEX-1:5|6:d
LIN
1:1d(2+1)2n
2:1o(4+1)3d
3:3d(2+1)4n
4:3o(4+1)5d
5:5o(3+1)6d
6:6o(2+1)7d
7:7d(2+1)8n
8:6o(4+1)9d
9:9d(2+1)10n
10:5o(6+1)11d
11:11o(2+1)12d
12:12d(2+1)13n
13:11o(6+1)14d
14:14d(2+1)15n
15:1o(6+1)16d

Here is the resulting csv file, which is sent to the "csv" subdirectory of the GlycoCT directory:

### base/GlycoCT/csv/G00176HZ.csv ###
glycan_ID,residue,residue_ID,name,anomer,absolute,ring,parent_ID,site,shape,color,pubChem
G00176HZ,unassigned,5,GlcNAc,b,D,p,4,2,square,#0090BC,GlcNAc
G00176HZ,unassigned,6,GlcNAc,b,D,p,4,4,square,#0090BC,GlcNAc
G00176HZ,unassigned,4,Man,a,D,p,3,3,circle,#00A651,Man
G00176HZ,unassigned,8,GlcNAc,b,D,p,7,2,square,#0090BC,GlcNAc
G00176HZ,unassigned,9,GlcNAc,b,D,p,7,6,square,#0090BC,GlcNAc
G00176HZ,unassigned,7,Man,a,D,p,3,6,circle,#00A651,Man
G00176HZ,unassigned,3,Man,b,D,p,2,4,circle,#00A651,Man
G00176HZ,unassigned,2,GlcNAc,b,D,p,1,4,square,#0090BC,GlcNAc
G00176HZ,unassigned,10,Fuc,a,L,p,1,6,triangle,#ED1C24,Fuc
G00176HZ,unassigned,1,GlcNAc,b,D,p,0,0,square,#0090BC,GlcNAc


###

Here is a command to convert all files IN A SPECIFIED LIST (base/GlycoCT/files.txt) to csv format (verbosity is zero)

$java -jar GCT2csv.jar base/GlycoCT/files.txt list 0

This is invoked from the (code) directory holding GCT2csv.jar file
The files.txt file can be generated from within the GlycoCT directory as follows:

$ls -1 G* > files.lst
$awk -f addPWD.awk files.lst > files.txt

files.txt contains lines like this:

$head -3 files.txt
base/GlycoCT/G00025AJ.txt
base/GlycoCT/G00031MO.txt
base/GlycoCT/G00033MO.txt

Note that not all monosaccharides are supported by GCT2csv, specifically 4 files in the above directory contain dxyl-HEX residues (not supported yet).  These can be identified using within the GlycoCT directory:

grep "dxyl-HEX" -l G*

which returns the following files:

G31249PS.txt    G32227EH.txt    G47349YO.txt    G58921NR.txt

Other partially defined structures should be removed before conversion to csv.  Most of these can be identified using the following command within the GlycoCT directory:

grep "([123456789]|" -l G*

This shows GlycoCT files with content like this:
G01028CW.txt:           5:5o(3|6+1)6d
Here the linkage can be to either O3 OR O6 - i.e., 3|6

GCT2csv.jar fails for these cases, so, PRIOR TO INVOKING GCT2csv.jar, these should be moved to the "movedOut" directory with the following command:

mv `grep "([123456789]|" -l G*` ./movedOut/

Here are the GlycoCT files in the movedOut directory at the time of this writing:

G01028CW.txt	G09430FY.txt	G18510WC.txt	G29539MT.txt	G36839GD.txt	G49124LP.txt	G62960MX.txt	G71041DQ.txt	G79976GJ.txt	G89144TN.txt   G01031EZ.txt	G10080RX.txt	G19710JA.txt	G30070XN.txt	G37320GX.txt	G49763BM.txt	G64085NR.txt	G71145QW.txt	G80357KJ.txt	G89661WR.txt   G01465MJ.txt	G10859BP.txt	G19863KG.txt	G30335ZU.txt	G37745TJ.txt	G50425IY.txt	G64384IL.txt	G72175XZ.txt G80434TI.txt	G90251SV.txt   G02000AU.txt	G11024UW.txt	G19934BY.txt	G30743FK.txt	G38473PC.txt	G51247NH.txt	G64652AR.txt	G73076IB.txt	G81911LP.txt	G90686TN.txt   G02297WO.txt	G11346ER.txt	G21573GX.txt	G30853JB.txt	G38909RH.txt	G51840MR.txt	G64852XI.txt  G73173YQ.txt	G82216JH.txt	G90999QY.txt  G02718AK.txt	G12899EM.txt	G21667GZ.txt	G31021LA.txt	G39305NP.txt	G52805ZP.txt	G65318OO.txt	G73408MO.txt	G82987BA.txt	G91410ZU.txt   G03137RO.txt	G12932IO.txt	G21998NE.txt	G31249PS.txt	G41477DH.txt	G53977IC.txt   G65909WP.txt	G73487DJ.txt	G84629NY.txt	G92232VN.txt G03965VN.txt	G13374TU.txt	G22048BB.txt	G31438WA.txt	G41742IZ.txt	G54553CA.txt	G66998OZ.txt	G73868PD.txt	G84882JT.txt	G93319SH.txt   G03987JH.txt	G13698FQ.txt	G22493AJ.txt	G31729QS.txt	G42140AQ.txt    G54602MP.txt	G67091KC.txt	G74714EN.txt	G85454WJ.txt	G93412AQ.txt   G04286YE.txt	G13757MF.txt	G22898AD.txt	G32227EH.txt	G43426MU.txt	G54873FX.txt	G67734FT.txt	G74838NI.txt	G85545QE.txt	G93450CE.txt    G04561MG.txt	G15826FL.txt	G23893ZL.txt	G32522KP.txt G44842OK.txt	G55037II.txt	G68146EG.txt	G75433IQ.txt	G85562TB.txt	G94449IF.txt   G05098FE.txt	G16144PA.txt	G24539CG.txt	G33419PP.txt	G45071EF.txt	G57210SQ.txt	G69062KW.txt	G77195RI.txt	G86114KZ.txt	G94615OV.txt   G05206VR.txt	G16175MV.txt	G24563OC.txt  G33866SM.txt	G45145ET.txt	G58921NR.txt	G69303AY.txt	G77964WB.txt	G86646IC.txt	G96681ZB.txt  G05233KP.txt	G16403CM.txt	G26450SH.txt	G34797BZ.txt	G45996VR.txt	G59874JZ.txt	G69808IK.txt	G78034SR.txt	G86718PI.txt	G96746BC.txt   G07429FY.txt	G17644VM.txt   G28023AU.txt	G35068DX.txt	G46926OL.txt	G61818QD.txt	G70688TC.txt	G78657XR.txt	G86874HJ.txt	G97413OV.txt G07681KR.txt	G17849ZC.txt	G28378XH.txt	G35242IF.txt	G47349YO.txt	G62088FQ.txt	G70829NO.txt	G79021WI.txt	G88140NN.txt	G97607IA.txt   G09425PN.txt	G18463FK.txt	G29081NM.txt	G35312VR.txt	G48265KY.txt	G62870HD.txt	G70848XM.txt	G79928VL.txt	G88687UE.txt


****** Mapping the csv files to a canonical N-gycan tree. *****

Complete mapping is only possible for fully-specified structure files.
Manually find fuzzy structures (e.g., linkage unknown) using:
    $grep -l ",-1," G*.csv > fuzzy.txt
The files listed in fuzzy.txt may require special attention.


The csv files (see above) are mapped to a canonical N-glycan tree using the java program called TreeBuilder2, which has a command-line syntax like this:

$java -jar TreeBuilder2.jar -g <csvFile> -s <sugarsFile> -c <N-nodesFile> -n 3 -v 5 -m 1 -e 5 -o <N-nodes-extentionFile>

-  The -g option specifies that a single glycan (csv) file should be processed. Alternatively, a list can be specified using the -l option, for example,
    TreeBuilder2.jar -l list1.txt ... ..., where list1.txt contains a list of fully specified glycan file paths
-  The -s option specifies a csv file containing information about SNFG sugars (shape, color, etc).
-  The -c option specifies a csv file containing the currently accepted canonical N-glycan nodes.
-  the -n option specifies the minimum length of a matching traversal to the root to accept the match.  
    For example, "-n 3" specifies that each matching path must have at least 3 residues, such that only glycans with a Man-GlcNac-GlcNAc core will match, and O-GlcNAc (monosaccharide) will not match to the root residue of the N-glycan tree.
-  The -v option specifies the verbosity of the output - set to 0 for large data lists, and higher to see how a particular glycan structure file is processed.
-  The -m option specifies the strictness of a match. Examples: 
    "-m 0" specifies that only exact matches will be accepted.
    "-m 1" specifies that only exact matches will be accepted, except the anomeric configuration of the root residue does not have to match - allows reducing glycan structures to match N-linked structures.
-  The -e option specifies how partially mismatched traversals are processed - writing new canonical residues to  extend the canonical list, with the extended canonical node list written to the file specified by the -o option
-  The -o option specifies the name of the file containing the extended canonical node list.

The list to be used above can be generated as described above, but invoked from the csv directory.
    
Here is a real example for a single input file:

$java -jar TreeBuilder2.jar -g base/GlycoCT/csv/G00176HZ.csv -s canonicalData/sugars.csv -c canonicalData/N-nodes.csv -n 3 -v 9 -m 1 -e 5 -o canonicalData/N-nodes-test.csv

Here, "canonicalData" is a symbolic link to a directory containing the relevant files.
The command gives the following result:

- A report file (report.csv) is generated in the "mapped" subdirectory.  This file includes information about criteria for matching or non-matching structures that have been processed.
- An extension of the canonical N-glycan tree is generated, and put into the file specified by the -o option. This can be checked for correctness and the vetted entities added to the N-nodes.csv file for subsequent mappings.
- A new structure file (mapped to the canonical tree defined in N-nodes.csv) for each specified glycan is generated, using the same file name but placed in a subdirectory called "mapped".

A REAL example:

$java -jar TreeBuilder2.jar -l base/GlycoCT/csv/files.txt -s canonicalData/sugars.csv -c canonicalData/N-nodes-full-2019-05-29.csv -n 3 -v 0 -m 1 -e 5 -o canonicalData/N-nodes-ext.csv

NOTE! This fails for files containing GlcN (NOT GlcNAc) with a Crossed Square shape... Use this command from within the csv directory:

$grep "Crossed Square" G*
G10713RP.csv:G10713RP,unassigned,1,GlcN,b,D,p,0,0,Crossed Square,#0090BC,GlcN
G12185WN.csv:G12185WN,unassigned,1,GlcN,a,D,p,0,0,Crossed Square,#0090BC,GlcN
G19577LJ.csv:G19577LJ,unassigned,1,GlcN,b,D,p,0,0,Crossed Square,#0090BC,GlcN
G23935WM.csv:G23935WM,unassigned,1,GlcN,b,D,p,0,0,Crossed Square,#0090BC,GlcN
G27839GC.csv:G27839GC,unassigned,2,GlcN,b,D,p,1,3,Crossed Square,#0090BC,GlcN
G27839GC.csv:G27839GC,unassigned,1,GlcN,b,D,p,0,0,Crossed Square,#0090BC,GlcN
G32586BA.csv:G32586BA,unassigned,1,GlcN,a,D,p,0,0,Crossed Square,#0090BC,GlcN
G47447OZ.csv:G47447OZ,unassigned,1,GlcN,a,D,p,0,0,Crossed Square,#0090BC,GlcN
G61071JD.csv:G61071JD,unassigned,2,GlcN,b,D,p,1,4,Crossed Square,#0090BC,GlcN
G61071JD.csv:G61071JD,unassigned,1,GlcN,b,D,p,0,0,Crossed Square,#0090BC,GlcN
G67665XP.csv:G67665XP,unassigned,1,GlcN,a,D,p,0,0,Crossed Square,#0090BC,GlcN
G78153TS.csv:G78153TS,unassigned,1,GlcN,a,D,p,0,0,Crossed Square,#0090BC,GlcN
G91079GH.csv:G91079GH,unassigned,1,GlcN,a,D,p,0,0,Crossed Square,#0090BC,GlcN
G91902KK.csv:G91902KK,unassigned,1,GlcN,a,D,p,0,0,Crossed Square,#0090BC,GlcN
G94923PG.csv:G94923PG,unassigned,1,GlcN,a,D,p,0,0,Crossed Square,#0090BC,GlcN
G99506BB.csv:G99506BB,unassigned,1,GlcN,a,D,p,0,0,Crossed Square,#0090BC,GlcN

When TreeBulder2 in invoked, Std Out gives error messages ... this bug can be fixed, but for now, these files are not important (These are not N-glycans)

This takes 2552 csv files in the input list and generates 1490 mapped csv files.
Many of these are in full agreement with the original (input) canonical node file, but many have only single terminal residues that do not match.  These files are mapped to matching (original) canonical residues and (in part) to (new) canonical residues that are saved in the N-nodes-ext.csv file. 

Residues whose linkage site or other features are undefined are not annotated (other than as "unassigned".)  Here is an example:

*** G95924JX.txt *** 
RES
1b:b-dglc-HEX-1:5
2s:n-acetyl
3b:b-dglc-HEX-1:5
4s:n-acetyl
5b:b-dman-HEX-1:5
6b:a-dman-HEX-1:5
7b:b-dglc-HEX-1:5
8s:n-acetyl
9b:b-dgal-HEX-1:5
10b:a-dgro-dgal-NON-2:6|1:a|2:keto|3:d
11s:n-acetyl
12b:b-dglc-HEX-1:5
13s:n-acetyl
14b:b-dgal-HEX-1:5
15b:a-dgro-dgal-NON-2:6|1:a|2:keto|3:d
16s:n-acetyl
17b:a-dman-HEX-1:5
18b:b-dglc-HEX-1:5
19s:n-acetyl
20b:b-dgal-HEX-1:5
21b:a-dgro-dgal-NON-2:6|1:a|2:keto|3:d
22s:n-acetyl
LIN
1:1d(2+1)2n
2:1o(4+1)3d
3:3d(2+1)4n
4:3o(4+1)5d
5:5o(3+1)6d
6:6o(2+1)7d
7:7d(2+1)8n
8:7o(4+1)9d
9:9o(-1+2)10d   // UNDEFINED LINKAGE SITE //
10:10d(5+1)11n
11:6o(4+1)12d
12:12d(2+1)13n
13:12o(4+1)14d
14:14o(-1+2)15d   // UNDEFINED LINKAGE SITE //
15:15d(5+1)16n
16:5o(6+1)17d
17:17o(2+1)18d
18:18d(2+1)19n
19:18o(4+1)20d
20:20o(3+2)21d
21:21d(5+1)22n

*** G95924JX.csv BEFORE MAPPING ***
glycan_ID,residue,residue_ID,name,anomer,absolute,ring,parent_ID,site,shape,color,pubChem
G95924JX,unassigned,7,NeuNAc,a,D,p,6,-1,diamond,#A54399,NeuNAc // UNASSIGNED LINKAGE SITE (site = -1)
G95924JX,unassigned,6,Gal,b,D,p,5,4,circle,#FFD400,Gal
G95924JX,unassigned,5,GlcNAc,b,D,p,4,2,square,#0090BC,GlcNAc
G95924JX,unassigned,10,NeuNAc,a,D,p,9,-1,diamond,#A54399,NeuNAc // UNASSIGNED LINKAGE SITE (site = -1)
G95924JX,unassigned,9,Gal,b,D,p,8,4,circle,#FFD400,Gal
G95924JX,unassigned,8,GlcNAc,b,D,p,4,4,square,#0090BC,GlcNAc
G95924JX,unassigned,4,Man,a,D,p,3,3,circle,#00A651,Man
G95924JX,unassigned,14,NeuNAc,a,D,p,13,3,diamond,#A54399,NeuNAc
G95924JX,unassigned,13,Gal,b,D,p,12,4,circle,#FFD400,Gal
G95924JX,unassigned,12,GlcNAc,b,D,p,11,2,square,#0090BC,GlcNAc
G95924JX,unassigned,11,Man,a,D,p,3,6,circle,#00A651,Man
G95924JX,unassigned,3,Man,b,D,p,2,4,circle,#00A651,Man
G95924JX,unassigned,2,GlcNAc,b,D,p,1,4,square,#0090BC,GlcNAc
G95924JX,unassigned,1,GlcNAc,b,D,p,0,0,square,#0090BC,GlcNAc


*** G95924JX.csv AFTER MAPPING ***
glycan_ID,residue,residue_ID,name,anomer,absolute,ring,parent_ID,site,shape,color,pubChem,comment
G95924JX,N-glycan_a-D-Manp_4,N4,Man,a,D,p,NC,6,circle,#00A651,Man
G95924JX,N-glycan_b-D-GlcpNAc_5,N5,GlcNAc,b,D,p,N4,2,square,#0090BC,GlcNAc
G95924JX,N-glycan_b-D-Galp_6,N6,Gal,b,D,p,N5,4,circle,#FFD400,Gal
G95924JX,N-glycan_a-D-Neup5Ac_16,N16,Neu5Ac,a,D,p,N6,3,diamond,#A54399,Neu5Ac
G95924JX,N-glycan_core_b-D-GlcpNAc_A,NA,GlcNAc,b,D,p,no_id,0,square,#0090BC,GlcNAc
G95924JX,N-glycan_core_b-D-GlcpNAc_B,NB,GlcNAc,b,D,p,NA,4,square,#0090BC,GlcNAc
G95924JX,N-glycan_core_b-D-Manp,NC,Man,b,D,p,NB,4,circle,#00A651,Man
G95924JX,N-glycan_a-D-Manp_1,N1,Man,a,D,p,NC,3,circle,#00A651,Man
G95924JX,N-glycan_b-D-GlcpNAc_2,N2,GlcNAc,b,D,p,N1,2,square,#0090BC,GlcNAc
G95924JX,N-glycan_b-D-Galp_3,N3,Gal,b,D,p,N2,4,circle,#FFD400,Gal
G95924JX,unassigned,7,Neu5Ac,a,D,p,N3,-1,diamond,#A54399,Neu5Ac  // UNASSIGNED LINKAGE SITE - NO ANNOTATION
G95924JX,N-glycan_b-D-GlcpNAc_7,N7,GlcNAc,b,D,p,N1,4,square,#0090BC,GlcNAc
G95924JX,N-glycan_b-D-Galp_8,N8,Gal,b,D,p,N7,4,circle,#FFD400,Gal
G95924JX,unassigned,10,Neu5Ac,a,D,p,N8,-1,diamond,#A54399,Neu5Ac  // UNASSIGNED LINKAGE SITE - NO ANNOTATION


***** sorting csv files *****

In order to simplify the process of implementing depth-first traversals of the csv files (each of which contains
a tree structure), the residues are sorted according to the linkage site (9th column), and the sorted files are 
put in a subdirectory (called "sorted") of the original file.

$./sortCSV.sh base/GlycoCT/csv/mapped 5

where mapped is the directory with files to be sorted and 5 is the verbosity

***** combining csv files to make a "master" mapping file *****

$awk -f combineCSV.awk base/GlycoCT/csv/mapped/sorted/G* > canonicalData/mapped-N-glycans.csv

This also orders the residues according to a depth-first traversal of the tree, which is required for mapping residues to the graphical rendering of the svg representation.