Q: How to create spark datasets from a file without using File reader I have a data file that has 4 data sections. Header data, Summary data, Detail data and Footer data. Each section has a fixed number of columns.Each section is divided by two rows that just have a single "#" as the row content.But different sections have different of columns. Is there a way I can avoid creating new files and just use spark tsv(tab seperated foramt) module or any other module to read the file into 4 datasets directly.If I read the file directly then I am loosing the extra columns in the next data section. It only reads the from the file only those columns as the first row of the file.
#deptno dname   location
10  Accounting  New York
20  Research    Dallas
30  Sales   Chicago
40  Operations  Boston
#
#
#grade  losal   hisal
1   700.00  1200.00
2   1201.00 1400.00
4   2001.00 3000.00
5   3001.00 99999.00
3   1401.00 2000.00
#
#
#ENAME  DNAME   JOB EMPNO   HIREDATE    LOC
ADAMS   RESEARCH    CLERK   7876    23-MAY-87   DALLAS
ALLEN   SALES   SALESMAN    7499    20-FEB-81   CHICAGO
BLAKE   SALES   MANAGER 7698    01-MAY-81   CHICAGO
CLARK   ACCOUNTING  MANAGER 7782    09-JUN-81   NEW YORK
FORD    RESEARCH    ANALYST 7902    03-DEC-81   DALLAS
JAMES   SALES   CLERK   7900    03-DEC-81   CHICAGO
JONES   RESEARCH    MANAGER 7566    02-APR-81   DALLAS
#
#
#Name    Age Address
Paul    23  1115 W Franklin
Bessy the Cow   5   Big Farm Way
Zeke    45  W Main St

Output:
Dataset d1 :
#deptno dname   location
10  Accounting  New York
20  Research    Dallas
30  Sales   Chicago
40  Operations  Boston

Dataset d2 :
#grade  losal   hisal
1   700.00  1200.00
2   1201.00 1400.00
4   2001.00 3000.00
5   3001.00 99999.00
3   1401.00 2000.00

Dataset d3 :
#ENAME  DNAME   JOB EMPNO   HIREDATE    LOC
ADAMS   RESEARCH    CLERK   7876    23-MAY-87   DALLAS
ALLEN   SALES   SALESMAN    7499    20-FEB-81   CHICAGO
BLAKE   SALES   MANAGER 7698    01-MAY-81   CHICAGO
CLARK   ACCOUNTING  MANAGER 7782    09-JUN-81   NEW YORK
FORD    RESEARCH    ANALYST 7902    03-DEC-81   DALLAS
JAMES   SALES   CLERK   7900    03-DEC-81   CHICAGO
JONES   RESEARCH    MANAGER 7566    02-APR-81   DALLAS

Dataset d4 :
#Name Age Address
Paul23  1115 W Franklin
Bessy the Cow   5   Big Farm Way
Zeke    45  W Main St

