Q: Pig script to calculate total, percentage along with group by Relatively new to Pig scripts. I have below script to derive the error details grouped by Error Code, Name and their respective count.
A = LOAD 'traffic_error_details.txt' USING 
    PigStorage(',')  as (id:int, error_code:chararray,error_name:chararray, error_status:int);
B = FOREACH A GENERATE A.error_code as errorCode,A.error_name as 
errorName,A.error_status as errorStatus;

C = GROUP B by ($0,$1,$2);
F = FOREACH C GENERATE group, COUNT(B) as count;
Dump F;

Above would give results as below :


*

*INVALID_PARAM,REQUEST_ERROR,10 

*INTERNAL_ERROR,SERVER_ERROR,15    

*NOT_ALLOWED,ACCESS_ERROR,4 

*UNKNOWN_ERR,UNKNOWN_ERROR,10

*NIL,NIL,11


I would want to display percentage of errors as well. So as below :


*

*INVALID_PARAM,REQUEST_ERROR,10,20%

*INTERNAL_ERROR,SERVER_ERROR,15,30%    

*NOT_ALLOWED,ACCESS_ERROR,4,9% 

*UNKNOWN_ERR,UNKNOWN_ERROR,10,20%

*NIL,NIL,11,21% 


Here total number of requests considered is 50. Out of which 21% are successful. Remaining are splitup of Error %.
So how to calculate the total as well in the same script and in the same tuple ? so that % could be calculated as (count/total)*100.
Total refers to the count of all records error_details.txt.

A: After you've gotten counts for each error code, you would need to do a GROUP ALL to find the total number of errors and add that field to every row. Then you can divide the error code counts by the total count to find percent. Make sure you convert the counts variables from type long to type double to avoid any integer division problems.
This is the code:
A = LOAD 'traffic_error_details.txt' USING PigStorage(',') as 
    (id:int, errorCode:chararray, errorName:chararray, errorStatus:int);
B = FOREACH A GENERATE errorCode, errorName, errorStatus;
C = GROUP B BY (errorCode, errorName, errorStatus);
D = FOREACH C GENERATE 
    FLATTEN(group) AS (errorCode, errorName, errorStatus),
    COUNT(B) AS num;
E = GROUP D ALL;
F = FOREACH E GENERATE 
    FLATTEN(D) AS (errorCode, errorName, errorStatus, num),
    SUM(D.num) AS num_total;
G = FOREACH F GENERATE 
    errorCode, 
    errorName, 
    errorStatus,
    num,
    (double)num/(double)num_total AS percent;

You'll notice I modified your code slightly. I grouped by (errorCode, errorName, errorStatus) instead of ($0,$1,$2). It's safer to refer to the field names themselves instead of their positions in case you modify your code in the future and the positions aren't the same.
