<p>A few years ago, I wrote a python script to convert FCS files into tab-separated format. It was far from handling all the possibilities that the format description offers, but at least it worked for some of the files produced on one of our machine: <a href="http://www.igh.cnrs.fr/equip/Seitz/en_equipe-programmes.html" rel="nofollow noreferrer">http://www.igh.cnrs.fr/equip/Seitz/en_equipe-programmes.html</a></p>

<p>The format documentation I found enabled decoding (see section 3 of the pdf you mention), but it requires reading data in binary mode.</p>

<p>The general idea of this format (and, I guess, many other binary formats), is that there is a header zone at the beginning of the file with a defined number of fields encoding numbers indicating how the rest of the file is structured. So a first phase is to parse this header, following the description given in the documentation of the format. The information extracted from the header tells where to find the data and how it is encoded, still according to rules described in the format documentation.</p>

<p>In case this may be useful, and for the record, here is the code from the above-mentioned script (after stripping comments, some of which are merely copied from the format documentation, and adding a few ones):</p>

<pre class="lang-python prettyprint-override"><code>#!/usr/bin/env python
"""This script tries to read FCS flow cytometry data.
Format parsing inspired by information found here:
http://isac-net.org/Resources-for-Cytometrists/Data-Standards/Data-File-Standards/Flow-Cytometry-Data-File-Format-Standards.aspx
"""

import re
# To decode binary-encoded data
import struct
import sys

class Parameter(object):
    """This object represents one of the parameter types that are present in a DATA segment of a FCS file."""
    __slots__ = ("p_name", "p_bits", "p_range", "p_ampl", "parser")
    def __init__(self, p_name, p_bits, p_range, p_ampl):
        self.p_name = p_name
        self.p_bits = p_bits
        self.p_range = p_range
        self.p_ampl = p_ampl
        # Function for parsing a value of the parameter in the data segment
        self.parser = None

##############################################
# Here starts the parsing of the header part #
# which tells where the other parts are.     #
##############################################

f = open(sys.argv[1], "rb")
# The format name is encoded in 6 letters
# An ASCII letter is coded with one octet
file_format = "".join([f.read(1) for __ in range(6)])
sys.stdout.write("Format: %s\n" % file_format)
# The format descriptions reserves 4 octets that we skip
skip = f.read(4)
# 8 octet chunks encode the start and end positions
# of different parts of the data
text_start = int(f.read(8).strip(" "))
text_end = int(f.read(8).strip(" "))
data_start = int(f.read(8).strip(" "))
data_end = int(f.read(8).strip(" "))
analysis_start = int(f.read(8).strip(" "))
analysis_end = int(f.read(8).strip(" "))

if (analysis_start and analysis_end):
    sys.stderr.write("Cannot deal with ANALYSIS segment of an FCS file.\n")

####################################################
# Here starts the parsing of the "TEXT" portion    #
# which describes how the data proper is organized #
####################################################
f.seek(text_start)
# The first character in the primary TEXT segment is the ASCII delimiter character.
sep = f.read(1)
if sep not in ["_", "@"]:
    alt_sep = "_@_"
elif sep not in ["_", "|"]:
    alt_sep = "_|_"
else:
    assert sep not in ["+", "|"]
    alt_sep = "+|+"
text_segment = f.read(text_end - text_start)

fields = text_segment.split(sep)

info = {}

i = 0
while i &lt; len(fields) - 1:
    key = fields[i]
    i += 1
    val = fields[i]
    i += 1
    # Keywords are case insensitive, they may be written in a file in lower case, upper case, or a
    # mixture of the two. However, an FCS file reader must ignore keyword case. A keyword value may
    # be in lower case, upper case or a mixture of the two. Keyword values are case sensitive.
    info[key.upper()] = val
print "%s events were detected." % info["$TOT"]
print "Each event is characterized by %s parameters" % info["$PAR"]

if info["$NEXTDATA"] != "0":
    sys.stderr.write("Some other data exist in the file but hasn't been parsed.\n")

# L - List mode. For each event, the value of each parameter is stored in the order in which the
# parameters are described. The number of bits reserved for parameter 1 is described using the
# $P1B keyword. There can be only one set of list mode data per data set. The $DATATYPE
# keyword describes the data format. This is the most versatile mode for the storage of flow
# cytometry data because mode C and mode U data can be created from mode L data.
assert info["$MODE"] == "L"

parameters = []

# indices of the parameters
p_indices = range(1, int(info["$PAR"]) + 1)
for i in p_indices:
    p_name = info["$P%dN" % i]
    p_bits =  info["$P%dB" % i]
    p_range = info["$P%dR" % i]
    p_ampl =  info["$P%dE" % i]
    parameters.append(Parameter(p_name, p_bits, p_range, p_ampl))

sys.stdout.write("The parameters are:\n%s\n" % "\t".join([par.p_name for par in parameters]))

# How are 32 bit words organized
if info["$BYTEORD"] == "4,3,2,1":
    endianness = "&gt;"
else:
    endianness = "&lt;"
    assert info["$BYTEORD"] == "1,2,3,4"

    # I stripped a long comment which is just a copy of the documentation
# Type of data:
if info["$DATATYPE"] == "I":
    for par in parameters:
        nb_bits = int(par.p_bits)
        assert nb_bits % 8 == 0
        nb_bytes = nb_bits / 8
        # Determine format string for unpacking (see https://docs.python.org/2/library/struct.html)
        if nb_bytes == 1:
            c_type = "B" # unsigned char
        elif nb_bytes == 2:
            c_type = "H" # unsigned short
        elif nb_bytes == 4:
            c_type = "L" # unsigned long
        elif nb_bytes == 8:
            c_type = "Q" # unsigned long long
        else:
            raise ValueError, "Number of bytes (%d) not valid for an integer (see https://docs.python.org/2/library/struct.html#byte-order-size-and-alignment)." % nb_bytes
        fmt = "%s%s" % (endianness, c_type)
        p_range = int(par.p_range)
        def parser(data):
            value = struct.unpack(fmt, data.read(nb_bytes))[0]
            try:
                assert value &lt; p_range
            except AssertionError:
                print "Value %s higher than %d" % (str(value), p_range)
            return value
        par.parser = parser
    pass
else:
    raise NotImplementedError, "Only the parsing of integer value has been implemented so far."


out_file = open(sys.argv[2], "w")
out_file.write("#amplification_types\t" + "\t".join([par.p_ampl for par in parameters]) + "\n")
out_file.write("parameters\t" + "\t".join([par.p_name for par in parameters]) + "\n")
i = 1
##############################################
# Here starts the parsing of the data proper #
##############################################
f.seek(data_start)
while f.tell() &lt; data_end:
    values = []
    for par in parameters:
        values.append(par.parser(f))
    out_file.write("%d\t" % i + "\t".join(map(str, values)) + "\n")
    i += 1
out_file.close()
f.close()
</code></pre>

CC BY-SA 3.0
2017-07-01T08:24:21.630Z
The general idea of this format (and, I guess, many other binary formats), is that there is a header zone at the beginning of the file with a defined number of fields encoding numbers indicating how the rest of the file is structured. So I begin by parsing this header, following the description given in the documentation of the format. The information extracted from the header tells me where to find the data and how it is encoded, still according to rules described in the format documentation. I'm not good at writing pseudocode, but I can try to add comments in the python code.
bli
CC BY-SA 3.0
2017-06-30T12:51:15.207Z
2017-07-04T07:01:36.560Z
2017-07-04T07:01:36.560Z
bli
<p>I maintain several Python libraries for interacting with flow cytometry data (FCS files). The most recent of which, <a href="https://github.com/whitews/flowkit" rel="nofollow noreferrer">FlowKit</a>, provides functionality to export the event data as a NumPy array, Pandas DataFrame, or CSV text. Using the <code>h5py</code> library, <a href="https://stackoverflow.com/questions/20928136/input-and-output-numpy-arrays-to-h5py">it's trivial to export as an HDF5 file</a></p>

CC BY-SA 4.0
2019-02-03T02:49:28.850Z
2019-02-03T02:49:28.850Z
Fiver
<p>R/Bioconductor has a number of different flow cytometry processing packages. One place to start for looking at cytometry data from a high level would be <a href="http://opencyto.org/" rel="nofollow noreferrer">openCyto</a> (or its <a href="http://opencyto.org/openCytoVignette.html" rel="nofollow noreferrer">vignette</a>), which is a large set of tools for basic extraction and analysis of FCS files.</p>

<p>I have looked in the past at the FCS files as an R structure using <a href="http://bioconductor.org/packages/release/bioc/html/flowCore.html" rel="nofollow noreferrer">flowCore</a>. Loading a single FCS file is fairly straightforward and follows a familiar R pattern:</p>

<pre><code>file.name &lt;- "/dir/file.fcs"
x &lt;- read.FCS(file.name, transformation=FALSE)
summary(x)
</code></pre>

<p>Asking about converting FCS files into an "easily readable format like HDF5" doesn't seem like the right question. <a href="https://en.wikipedia.org/wiki/Hierarchical_Data_Format#HDF5" rel="nofollow noreferrer">HDF5</a> is a container format and shares a lot of similarity to file systems. I've found it best to keep FCS files as they are, as it is a compact, standardised binary format.</p>

CC BY-SA 3.0
2017-07-01T07:23:54.767Z
HDF5 is not a major concern (I just like it). I am still curious about how something like `read.FCS` works. I believe if FCS is a standard format then there should be standard (non-proprietary) libraries to read/write FCS files. As of now I'm forced to use R or Matlab (and still there are no standard libraries here too. I guess there are just different packages/programs made by different groups/individuals; no single standard). What if I want to analyse a FCS file on C, Fortran or Python (etc)?
WYSIWYG
CC BY-SA 3.0
2017-06-30T11:56:45.483Z
2017-06-30T11:56:45.483Z
gringer
<p>FCS is a patented data format used for storing flow cytometry data. The most recent version is FCS3.1. There is some <a href="http://isac-net.org/PDFS/90/9090600d-19be-460d-83fc-f8a8b004e0f9.pdf" rel="noreferrer">documentation</a> on the format, but there is no information on how to read these files. There are some R packages and a MATLAB code to read an FCS file, but I am looking for standard libraries developed either by the FCS consortium or any other group. I also wish to know if FCS is a subset of an existing standard data format that can be read by a standard library using any programming language.</p>

<p>Finally, I would want to convert these files to an easily readable format like HDF5. </p>

CC BY-SA 3.0
2017-06-30T14:19:41.753Z
I find strange you ask for standard libraries but you exclude the existing standard tools.
llrs
CC BY-SA 3.0
2017-06-30T14:28:25.690Z
@Llopis the OP isn't rejecting standard tools, on the contrary, he is looking for standard libraries.
terdon
CC BY-SA 3.0
2017-06-30T14:32:37.387Z
@terdon I am not sure how standard libraries are decided but a R/Bioconductor package or a MATLAB one seems to me quite standard (Except if the FCS consortium provides one of course).
llrs
CC BY-SA 3.0
2017-07-01T06:28:02.557Z
@Llopis By standard, I mean is there an OS/language independent interface that allows reading of these files. More importantly, I want to know how FCS files can be read by any general language. I am expecting something like a pseudocode, for this purpose.
WYSIWYG
CC BY-SA 3.0
2017-07-01T12:13:12.683Z
I’m still not really sure what you want: the documentation you link to is as close to a general pseudocode as you will get: it contains a detailed, concise specification of the format. That’s the same information that a pseudocode would convey, and it’s the conventional way in which data formats are described (pseudocode is virtually never used for this). That, in addition to the existence of libraries for common analysis languages (R, Python, Matlab), should be sufficient to work with the format.
Konrad Rudolph
CC BY-SA 3.0
2017-07-01T13:54:40.923Z
@WYSIWYG In addition to what Konrad said, the flowCore in  Bioconductor is available for Windows, iOS and Linux.
llrs
CC BY-SA 3.0
2017-07-02T11:20:05.323Z
@KonradRudolph What I want is an instruction on how these files are to be read. The documentation describes the format but I could not figure out the general procedure to read the format which is what I meant by pseudocode. Just assume that I **do not** want to use bioconductor;  lets say that I want to read these files in C. How should I go about that? Do the FACS machines use a standard mechanism/io library for reading/writing FCS files or they use their own codes? I assumed that if a format is defined by a consortium, they would have provided a standard interface for rw operation.
WYSIWYG
CC BY-SA 3.0
2017-07-04T16:11:04.907Z
Why do you say .fcs is a patented format? The disclaimer in the linked documentation just states "Attention is called to the possibility that implementation of this standard may require use of subject matter covered by patent rights".
It may not even be possible to patent it, see: https://patents.stackexchange.com/questions/11968/can-patents-on-computer-file-formats-be-obtained
Peter
<file-formats>
How can I read FCS files using open source libraries?
# How can I read FCS files using open source libraries?

(Asked by: username_0 on 2017-06-30)

FCS is a patented data format used for storing flow cytometry data. The most recent version is FCS3.1. There is some [documentation](http://isac-net.org/PDFS/90/9090600d-19be-460d-83fc-f8a8b004e0f9.pdf) on the format, but there is no information on how to read these files. There are some R packages and a MATLAB code to read an FCS file, but I am looking for standard libraries developed either by the FCS consortium or any other group. I also wish to know if FCS is a subset of an existing standard data format that can be read by a standard library using any programming language.

Finally, I would want to convert these files to an easily readable format like HDF5.

**Tags:** file-formats
> **Comments:**
> **username_1**: I find strange you ask for standard libraries but you exclude the existing standard tools.  
> **username_2**: @Llopis the OP isn't rejecting standard tools, on the contrary, he is looking for standard libraries.  
> **username_1**: @terdon I am not sure how standard libraries are decided but a R/Bioconductor package or a MATLAB one seems to me quite standard (Except if the FCS consortium provides one of course).  
> **username_0**: @Llopis By standard, I mean is there an OS/language independent interface that allows reading of these files. More importantly, I want to know how FCS files can be read by any general language. I am expecting something like a pseudocode, for this purpose.  
> **username_3**: I’m still not really sure what you want: the documentation you link to is as close to a general pseudocode as you will get: it contains a detailed, concise specification of the format. That’s the same information that a pseudocode would convey, and it’s the conventional way in which data formats are described (pseudocode is virtually never used for this). That, in addition to the existence of libraries for common analysis languages (R, Python, Matlab), should be sufficient to work with the format.  
> **username_1**: @WYSIWYG In addition to what Konrad said, the flowCore in  Bioconductor is available for Windows, iOS and Linux.  
> **username_0**: @KonradRudolph What I want is an instruction on how these files are to be read. The documentation describes the format but I could not figure out the general procedure to read the format which is what I meant by pseudocode. Just assume that I **do not** want to use bioconductor;  lets say that I want to read these files in C. How should I go about that? Do the FACS machines use a standard mechanism/io library for reading/writing FCS files or they use their own codes? I assumed that if a format is defined by a consortium, they would have provided a standard interface for rw operation.  
> **username_4**: Why do you say .fcs is a patented format? The disclaimer in the linked documentation just states "Attention is called to the possibility that implementation of this standard may require use of subject matter covered by patent rights". It may not even be possible to patent it, see: https://patents.stackexchange.com/questions/11968/can-patents-on-computer-file-formats-be-obtained  

---

## 3 answers

### Answer by username_5 on 2017-06-30. Score: 5

A few years ago, I wrote a python script to convert FCS files into tab-separated format. It was far from handling all the possibilities that the format description offers, but at least it worked for some of the files produced on one of our machine: http://www.igh.cnrs.fr/equip/Seitz/en_equipe-programmes.html
The format documentation I found enabled decoding (see section 3 of the pdf you mention), but it requires reading data in binary mode.
The general idea of this format (and, I guess, many other binary formats), is that there is a header zone at the beginning of the file with a defined number of fields encoding numbers indicating how the rest of the file is structured. So a first phase is to parse this header, following the description given in the documentation of the format. The information extracted from the header tells where to find the data and how it is encoded, still according to rules described in the format documentation.
In case this may be useful, and for the record, here is the code from the above-mentioned script (after stripping comments, some of which are merely copied from the format documentation, and adding a few ones):
#!/usr/bin/env python
"""This script tries to read FCS flow cytometry data.
Format parsing inspired by information found here:
http://isac-net.org/Resources-for-Cytometrists/Data-Standards/Data-File-Standards/Flow-Cytometry-Data-File-Format-Standards.aspx
"""

import re
# To decode binary-encoded data
import struct
import sys

class Parameter(object):
    """This object represents one of the parameter types that are present in a DATA segment of a FCS file."""
    __slots__ = ("p_name", "p_bits", "p_range", "p_ampl", "parser")
    def __init__(self, p_name, p_bits, p_range, p_ampl):
        self.p_name = p_name
        self.p_bits = p_bits
        self.p_range = p_range
        self.p_ampl = p_ampl
        # Function for parsing a value of the parameter in the data segment
        self.parser = None

##############################################
# Here starts the parsing of the header part #
# which tells where the other parts are.     #
##############################################

f = open(sys.argv[1], "rb")
# The format name is encoded in 6 letters
# An ASCII letter is coded with one octet
file_format = "".join([f.read(1) for __ in range(6)])
sys.stdout.write("Format: %s\n" % file_format)
# The format descriptions reserves 4 octets that we skip
skip = f.read(4)
# 8 octet chunks encode the start and end positions
# of different parts of the data
text_start = int(f.read(8).strip(" "))
text_end = int(f.read(8).strip(" "))
data_start = int(f.read(8).strip(" "))
data_end = int(f.read(8).strip(" "))
analysis_start = int(f.read(8).strip(" "))
analysis_end = int(f.read(8).strip(" "))

if (analysis_start and analysis_end):
    sys.stderr.write("Cannot deal with ANALYSIS segment of an FCS file.\n")

####################################################
# Here starts the parsing of the "TEXT" portion    #
# which describes how the data proper is organized #
####################################################
f.seek(text_start)
# The first character in the primary TEXT segment is the ASCII delimiter character.
sep = f.read(1)
if sep not in ["_", "@"]:
    alt_sep = "_@_"
elif sep not in ["_", "|"]:
    alt_sep = "_|_"
else:
    assert sep not in ["+", "|"]
    alt_sep = "+|+"
text_segment = f.read(text_end - text_start)

fields = text_segment.split(sep)

info = {}

i = 0
while i < len(fields) - 1:
    key = fields[i]
    i += 1
    val = fields[i]
    i += 1
    # Keywords are case insensitive, they may be written in a file in lower case, upper case, or a
    # mixture of the two. However, an FCS file reader must ignore keyword case. A keyword value may
    # be in lower case, upper case or a mixture of the two. Keyword values are case sensitive.
    info[key.upper()] = val
print "%s events were detected." % info["$TOT"]
print "Each event is characterized by %s parameters" % info["$PAR"]

if info["$NEXTDATA"] != "0":
    sys.stderr.write("Some other data exist in the file but hasn't been parsed.\n")

# L - List mode. For each event, the value of each parameter is stored in the order in which the
# parameters are described. The number of bits reserved for parameter 1 is described using the
# $P1B keyword. There can be only one set of list mode data per data set. The $DATATYPE
# keyword describes the data format. This is the most versatile mode for the storage of flow
# cytometry data because mode C and mode U data can be created from mode L data.
assert info["$MODE"] == "L"

parameters = []

# indices of the parameters
p_indices = range(1, int(info["$PAR"]) + 1)
for i in p_indices:
    p_name = info["$P%dN" % i]
    p_bits =  info["$P%dB" % i]
    p_range = info["$P%dR" % i]
    p_ampl =  info["$P%dE" % i]
    parameters.append(Parameter(p_name, p_bits, p_range, p_ampl))

sys.stdout.write("The parameters are:\n%s\n" % "\t".join([par.p_name for par in parameters]))

# How are 32 bit words organized
if info["$BYTEORD"] == "4,3,2,1":
    endianness = ">"
else:
    endianness = "<"
    assert info["$BYTEORD"] == "1,2,3,4"

    # I stripped a long comment which is just a copy of the documentation
# Type of data:
if info["$DATATYPE"] == "I":
    for par in parameters:
        nb_bits = int(par.p_bits)
        assert nb_bits % 8 == 0
        nb_bytes = nb_bits / 8
        # Determine format string for unpacking (see https://docs.python.org/2/library/struct.html)
        if nb_bytes == 1:
            c_type = "B" # unsigned char
        elif nb_bytes == 2:
            c_type = "H" # unsigned short
        elif nb_bytes == 4:
            c_type = "L" # unsigned long
        elif nb_bytes == 8:
            c_type = "Q" # unsigned long long
        else:
            raise ValueError, "Number of bytes (%d) not valid for an integer (see https://docs.python.org/2/library/struct.html#byte-order-size-and-alignment)." % nb_bytes
        fmt = "%s%s" % (endianness, c_type)
        p_range = int(par.p_range)
        def parser(data):
            value = struct.unpack(fmt, data.read(nb_bytes))[0]
            try:
                assert value < p_range
            except AssertionError:
                print "Value %s higher than %d" % (str(value), p_range)
            return value
        par.parser = parser
    pass
else:
    raise NotImplementedError, "Only the parsing of integer value has been implemented so far."


out_file = open(sys.argv[2], "w")
out_file.write("#amplification_types\t" + "\t".join([par.p_ampl for par in parameters]) + "\n")
out_file.write("parameters\t" + "\t".join([par.p_name for par in parameters]) + "\n")
i = 1
##############################################
# Here starts the parsing of the data proper #
##############################################
f.seek(data_start)
while f.tell() < data_end:
    values = []
    for par in parameters:
        values.append(par.parser(f))
    out_file.write("%d\t" % i + "\t".join(map(str, values)) + "\n")
    i += 1
out_file.close()
f.close()

> **Comments:**
> **username_5**: The general idea of this format (and, I guess, many other binary formats), is that there is a header zone at the beginning of the file with a defined number of fields encoding numbers indicating how the rest of the file is structured. So I begin by parsing this header, following the description given in the documentation of the format. The information extracted from the header tells me where to find the data and how it is encoded, still according to rules described in the format documentation. I'm not good at writing pseudocode, but I can try to add comments in the python code.  

---

### Answer by username_6 on 2019-02-03. Score: 4

I maintain several Python libraries for interacting with flow cytometry data (FCS files). The most recent of which, [FlowKit](https://github.com/whitews/flowkit), provides functionality to export the event data as a NumPy array, Pandas DataFrame, or CSV text. Using the `h5py` library, [it's trivial to export as an HDF5 file](https://stackoverflow.com/questions/20928136/input-and-output-numpy-arrays-to-h5py)


---

### Answer by username_7 on 2017-06-30. Score: 3

R/Bioconductor has a number of different flow cytometry processing packages. One place to start for looking at cytometry data from a high level would be openCyto (or its vignette), which is a large set of tools for basic extraction and analysis of FCS files.
I have looked in the past at the FCS files as an R structure using flowCore. Loading a single FCS file is fairly straightforward and follows a familiar R pattern:
file.name <- "/dir/file.fcs"
x <- read.FCS(file.name, transformation=FALSE)
summary(x)

Asking about converting FCS files into an "easily readable format like HDF5" doesn't seem like the right question. HDF5 is a container format and shares a lot of similarity to file systems. I've found it best to keep FCS files as they are, as it is a compact, standardised binary format.

> **Comments:**
> **username_0**: HDF5 is not a major concern (I just like it). I am still curious about how something like `read.FCS` works. I believe if FCS is a standard format then there should be standard (non-proprietary) libraries to read/write FCS files. As of now I'm forced to use R or Matlab (and still there are no standard libraries here too. I guess there are just different packages/programs made by different groups/individuals; no single standard). What if I want to analyse a FCS file on C, Fortran or Python (etc)?  

---

