http___fmi.wikidot.com_os5_1.xml
http___www.enterprisestorageforum.com_storage-management_object-vs.-posix-storage-2.html_2.xml
1.1 2.1 3.1 4.1 5.1 5.2 6.1 6.2 7.1 8.1 8.2 8.3 8.4 8.5 9.1 10.1 11.1 12.1 12.2 12.3 12.4 13.1 13.2 13.3 14.1 15.1 16.1 17.1 18.1 18.2 18.3 19.1 20.1 21.1 22.1 23.1 23.2 23.3 24.1 25.1 26.1 26.2 26.3 27.1 28.1 29.1 30.1 30.2 31.1 32.1 33.1 34.1 35.1 36.1 37.1 38.1 39.1 39.2 40.1 41.1 42.1 43.1 44.1 45.1 46.1 46.2 46.3 47.1 48.1 48.2 49.1 50.1 50.2 50.3 51.1 52.1 53.1 54.1 55.1 56.1 57.1 58.1 59.1 60.1 61.1 62.1 63.1 64.1 65.1 65.2 66.1 67.1 68.1 69.1 70.1 71.1 72.1 73.1 75.1 76.1 77.1 78.1 79.1 81.1 82.1 84.1 84.2 85.1 87.1 88.1 90.1 91.1 93.1 94.1 97.1 98.1 99.1 100.1 101.1 103.1 104.1 105.1 106.1 107.1 109.1 110.1 111.1 112.1 113.1 113.2 114.1 115.1 116.1 117.1 118.1 119.1 120.1 121.1 124.1 125.1
Системни примитиви за работа с файлове. страницата се нуждае от дописване/преглеждане, щото някои работи не са ми съвсем ясни Презентацията на преподавателя http://newkis.fmi.uni-sofia.bg/~moni/osKN_ppt/fsys2_SC.ppt Важна част от работата на всяка операционна система е работата с файлове, и затова всяка ОС има средства за това. Тези средства са системни примитиви (системни извиквания (system calls), системни функции) на съответната система. В стандарта POSIX (Portable Operating System Interface for Unix) са дефинирани системните примитиви за достъп до файлове за тези операционни системи, които се придържат към него1 - например MINIX, UNIX, Linux, Solaris, FreeBSD и др. Ще разгледаме някои основни POSIX понятия и команди. Основни понятия - Файлов дескриптор - Неотрицателно цяло число, което служи за уникален идентификатор на отворен файл. При отваряне на файл, ядрото извършва необходимите действия за отваряне на файла, зарежда информация за него. След това връща файловия дескриптор, който се свързва с отворения файл, ползва се в програмата, а след затваряне се освобождава. Дескрипторът има локално значение - връзката между дескриптор (число) и отворен файл важи само за текущия процес. Всеки процес стандартно има отворени три дескриптора: - 0 (стандартен вход); - 1 (стандартен изход); - 2 (стандартен изход за грешки). - Текуща позиция във файл (file offset, file pointer) - Определя позицията на файла за четене и писане (един указател за двете действия). От гледна точка на ядрото файлът представлява масив от байтове. Отместването указва броя байтове от началото на файла до текущата позиция. Всеки файлов дескриптор има собствена позиция, която се променя от извършваните върху него процедури за четене и писане. - Режим на отваряне - Указва режима, в който е отворен даден дескриптор. При опит за четене / писане, в зависимост от режима, операцията се позволява или забранява. Основните режими са четене, писане, четене и писане. Системни примитиви за работа с обикновен файл Ще разгледаме някои функции във вариант за C/C++. Създаване на файл int creat(const char *filename, mode_t mode); Създава обикновен файл (не е специален - устройство, или връзка - link) с име filename в режим mode. Името на файла може да е абсолютно или относително. Режимът представлява съкратен код на защитата (permissions на потребителите за този файл) и се указва с 12-битово двоично число, което може да бъде записано като 4-цифрено осмично. mode_t е тип, дефиниран в хедъра   - така се осигурява по-голяма междуплатформена съвместимост. Връща дескриптора на файла, за който се назначава най-ниското свободно число, или -1 (невалиден дескриптор) при грешка. Отваряне на файл #include int open(const char *filename, int oflag [, mode_t mode]); Отваря файла filename в режим на отваряне oflag, а ако не съществува, го създава в режим mode. Създаването е позволено в по-нови версии на UNIX/Linux, в които creat() е запазена само за съвместимост. Така следните две са еквивалентни: creat(path, mode) open(path, O_WRONLY|O_CREAT|O_TRUNC, mode) oflag указва режима на отваряне и/или действия, които да се извършат след отваряне. Може да се зададе със символични константи, дефинирани в хедъра fcntl.h. Част от тях: - O_RDONLY - само за четене; - O_WRONLY - само за писане; - O_RDWR - за четене и писане; Едно от тези е задължително. Към него чрез побитово или може да се добавят някои от следните: - O_CREAT - създава файла; - O_EXCL - заедно с O_CREAT: ако файлът не съществува, той се създава, иначе се връща грешка; - O_TRUNC - старото съдържание на файла се изтрива след отваряне; - O_APPEND - старото съдържание се запазва, а новото се вписва в края на файла (при писане, текущата позиция автоматично отива в края на файла); - O_SYNC - всяко писане е синхронно (за обикновени файлове, по подразбиране се използва delayed write - данните се записват в буферния кеш, а ядрото се грижи за тяхното действително записване). Връща дескриптора на файла или -1 при грешка. Затваряне на файл int close(int fd); Затваря файла и освобождава файловия дескриптор. Връща 0 при успех и -1 при грешка. Четене и писане във файл ssize_t read(int fd, void *buffer, size_t nbytes); ssize_t write(int fd, void *buffer, size_t nbytes); - ssize_t - разновидност на познатия ни size_t, но със знак; - fd - номер на файлов дескриптор; - buffer - указател към програмния буфер - областта от програмата, където се записват данните (void* е сложено за универсалност); - nbytes - указва броя байтове за четене / писане. Четенето / писането започва от текущата позиция във файла. Може тя да е след края на файла, което не се счита за грешка. - При четене след края на файла, връща се 0. - При писане след края на файла: ако е указан режим O_APPEND, писането се извършва в края на файла. В противен случай, байтовете се пишат на текущата позиция, а размерът на файла се увеличава с броя записани байтове плюс размера на "дупката" между тях и EOF - т. нар. файлове с дупки Файл с дупки: При указан режим O_SYNC write() връща управление чак след действителното дисково записване; иначе, ползва се delayed write (виж по-горе). При O_SYNC, целият файл се презаписва на диска, независимо каква част от него е променена, което не е ефективно. Режимът се ползва за сигурен запис. Връща се броят действително прочетени / записани байтове (може да е по-малък от nbytes), 0 при EOF, -1 при грешка. Позициониране във файл off_t lseek(int fd, off_t offset, int flag); off_t обикновено указва long. offset е самото отместване. flag указва как се интерпретира отместването, за да се получи новият file pointer: - SEEK_SET (стойност 0): fp = offset - SEEK_CUR (стойност 1): fp = fp + offset - SEEK_END (стойност 2): fp = file_size + offset (да, "+" е - внимавайте!) Най-бърз system call - не се извършват дискови операции и не се променя размерът на файла. lseek(file, 0, 1) връща текущата позиция във file. Връща новата позиция при успех. Информация за файл #include int stat(const char * filename,struct stat* sbuf); int fstat(int fd, struct stat* sbuf); filename е знаков низ - име на файла. fd е файлов дескриптор. stat е структура, дефинирана в хедъра. Елементите `и придобиват и връщат атрибутите на файла. Връща 0 при успех, както и стойностите в структурата, -1 при неуспех (например несъществуващ файл). Примерни процедури Създава ново копие на съществуващ файл, чието име е зададено като първи аргумент, а името на създавания файл, като втори. #include #include #include #define BUFS 4096 main(int argc, char *argv[]) int fdr, fdw, n; struct stat sbuf; char buff[BUFS]; if (argc != 3) { fprintf(stderr, "usage: copy from_file to_file\n"); if ( stat(argv[1], &sbuf) == -1) { fprintf(stderr, "copy: %s: No such file\n", argv[1]); if ( ! S_ISREG(sbuf.st_mode) ) { fprintf(stderr, "copy: %s: Not a regular file\n", argv[1]); if (( fdr = open(argv[1], O_RDONLY)) == -1) { fprintf(stderr, "copy: %s: can't open\n", argv[1]); if (( fdw = creat(argv[2], 0640)) == -1) { fprintf(stderr, "copy: %s:can't create\n", argv[2]); while (( n = read(fdr, buff, BUFS)) > 0) write(fdw, buff, n); Извежда на стандартния изход съдържанието на файлове, чиито имена са зададени като аргументи. #include #include #include main(int argc, char *argv[]) int fdr, i; struct stat sbuf; char buff; if (argc < 2) { fprintf(stderr,"usage: typef file...\n"); for ( i=1; i<argc; i++ ) { if ( stat(argv[i], &sbuf) ==-1) { fprintf(stderr,"typef: %s: No such file\n", argv[i]); continue; } if ( ! S_ISREG(sbuf.st_mode) ) { fprintf(stderr,"typef: %s: is not a regular file\n", argv[i]); continue; } if (( fdr = open(argv[i], O_RDONLY)) == -1) { fprintf(stderr, "typef: %s: can't open\n", argv[i]); continue; } while ( read(fdr, &buf, 1) ) write(1, &buf, 1); close(fdr); Допълнителни неща SUS (Single UNIX Specification), "наследникът" на POSIX.
1.1 2.1 2.2 2.3 2.4 3.1 4.1 5.1 6.1 7.1 8.1 9.1 9.2 10.1 10.2 11.1 11.2 11.3 11.4 11.5 12.1 13.1 13.2 14.1 14.2 14.3 15.1 15.2 15.3 15.4 15.5 15.6 16.1 17.1 17.2 17.3 17.4 17.5 18.1 19.1 20.1 21.1 22.1 23.1 24.1 25.1 26.1 27.1 28.1 28.2 29.1 29.2 29.3 29.4 30.1 30.2 31.1 31.2 32.1 33.1 33.2 33.3 33.4 33.5 34.1 35.1 36.1 37.1 38.1 39.1 40.1 40.2 40.3 40.4 40.5 40.6 41.1 42.1 42.2 44.1 45.1 46.1 47.1 47.2 48.1 48.2 49.1 49.2 49.3 49.4 50.1 50.2 51.1 51.2 52.1 52.2 53.1 54.1 55.1 55.2 56.1 57.1 58.1 58.2 58.3 60.1 61.1 62.1 62.2 63.1 63.2 64.1 65.1 65.2 66.1 67.1 68.1 69.1 69.2 70.1 70.2 70.3 71.1 71.2 72.1 72.2 72.3 72.4 72.5 73.1 74.1 75.1 76.1 77.1 78.1 79.1 80.1 80.2 81.1 81.2 81.3 82.1 82.2 82.3 82.4 83.1 83.2 83.3 83.4 83.5 83.6 84.1 85.1 86.1 86.2 86.3 87.1 87.2 87.3 87.4 88.1 88.2 88.3 89.1
Download the authoritative guide: Enterprise Data Storage 2018: Optimizing Your Storage Infrastructure Object-oriented storage is becoming very popular. It's being used for websites, medical records, databases, Hadoop, general data media storage, and more. One of the big draws of object oriented storage is its simplicity. There are only a few commands for object based file systems: - PUT (basically a "write" — PUT the object into the storage) - GET (basically a "read" — GET the object from the storage) - DELETE (delete the object which is the file) - POST (dds an object using HTML forms — similar to PUT but using HTML) - HEAD (returns an object's metadata but not the data itself) The POST and HEAD commands are not universal, but you do see them in object storage systems that are compatible with AWS S3. On the other hand, tried-and-true POSIX file systems that have a massive range of data access functions. Below is a partial list of IO functions for Linux: This is only a partial list of POSIX IO functions including some of the extended attributes. There are many more. Object-oriented storage systems are really an all-or-nothing situation. You can't do any byte-level file access like you can do with POSIX. When you read a file, you get the entire file via the GET function. Your application has to handle the data in the file. This means you have to port or rewrite your application to adapt to the storage paradigm. Intermediaries There are some object-storage intermediaries that provide a gateway to/from the object-storage system and POSIX-compliant applications. Examples include the following: These solutions GET/PUT complete files from the object storage to a local file system. The POSIX applications interact with the data on the local file system (which is POSIX-compliant). The intermediaries then copy the data to/from the object storage systems. The concept of the intermediaries is good, but the devil is always in the details. How often do you synchronize data from the local file system to the object storage? What if the POSIX application is doing lots of small byte IO with the data files on the local storage? How and when does the intermediary move the data to/from the object storage? What happens if the local storage crashes? How is data consistency insured? There are many, many more questions about these intermediaries, but in many cases they are the best solution available to handle data movement to/from the object storage for you. One of the more fully featured tools is s3ql. It acts as an intermediate file system between S3 and local applications. It is POSIX-compliant so applications can run using the storage without worry. It handles hardlinks, symlinks, standard permissions, extended attributes and file sizes up to 2TB. It also has some additional features unique to s3ql, including the following: - Encryption - Compression - De-duplication - Copy-on-Write/Snapshotting - Dynamic Size - Transparency - Immutable trees - Range of backends - Caching What Is Needed? Object storage systems are popular for many reasons, most notably simplicity, ease of use, and cost (at least compared to NAS systems). On the other hand, people complain that applications have to be rewritten to use the storage and that they are slower than other forms of storage. POSIX file systems are the most common storage system in use today. The POSIX compliance provides a wide range of IO functions for applications to use, including byte-level access. However, with the large number of IO functions comes complexity, both for the application and the file system. Typically POSIX storage is faster than object storage but also a bit more expensive. As a result, people using object storage want faster performance and byte-level access while retaining the simplicity, ease of use, and the cost. People using POSIX storage would like things to be simpler, approaching that of object storage, and they would like the cost to come down. At this point, I would normally insert a picture of the rainbow unikitty with butterfly wings and say good luck. However, I think it might be possible to combine object-storage and POSIX to create something that comes close to the ideal. Object-POSIX? What really is needed is a hybrid file system that contains some of the aspects of object storage and some of the aspects of a POSIX file system. An obvious starting point is to either add the object storage functions (GET/PUT/DELETE) to POSIX, or to add some POSIX functions to object storage. In either case, the interesting part is trying to find the balance between object-oriented storage and POSIX. Adding too many functions to object storage makes the result look much more like POSIX. Conversely, if not enough functions are added to the object storage, then it won't be as easy to port applications. For either approach the overall general goals, at least in my mind, should be the following. - Simple to use - Only a few additional IO functions - Allow easy porting of POSIX IO applications - Add byte-level access in some fashion to object storage or object storage functions to POSIX storage. The controversial addition is the last one — adding byte-level access to object storage or adding object functions to POSIX. In my opinion, and I'm sure there will be people who won't agree with it, it is better to start with the simple object storage and add the needed functions to it. The biggest reason for this choice is simplicity. Object storage is very simple in concept with a very few IO functions. In addition, while people have been sold on the idea of object storage, they have found that writing applications to use it directly is really not easy. Therefore, I think it's better to start with object storage and add a few useful byte-level functions so that it's infinitely easier to port applications to use object storage. Here is my list of initial functions to add to object storage. There are some specific reasons I chose just these five functions and not more (or less). If byte-level access is going to be added to object storage an "open" function will be needed that returns a file descriptor that can be used throughput the application. To get an idea of what this function entails, an excerpt of the POSIX "open" manpage from Linux is listed below: int open(const char *pathname, int flags); int open(const char *pathname, int flags, mode_t mode); int creat(const char *pathname, mode_t mode); Given a pathname for a file, open() returns a file descriptor, a small, nonnegative integer for use in subsequent system calls (read(2), write(2), lseek(2), fcntl(2), etc.). The file descriptor returned by a successful call will be the lowest-numbered file descriptor not currently open for the process. By default, the new file descriptor is set to remain open across an execve(2) (i.e., the FD_CLOEXEC file descriptor flag described in fcntl(2) is initially disabled; the O_CLOEXEC flag, described below, can be used to change this default). The file offset is set to the beginning of the file (see lseek(2)). A call to open() creates a new open file description, an entry in the system-wide table of open files. This entry records the file offset and the file status flags (modifiable via the fcntl(2) F_SETFL operation). A file descriptor is a reference to one of these entries; this reference is unaffected if pathname is subsequently removed or modified to refer to a different file. The new open file description is initially not shared with any other process, but sharing may arise via fork(2). The argument flags must include one of the following access modes: O_RDONLY, O_WRONLY, or O_RDWR. These request opening the file read-only, write-only, or read/write, respectively. The "open" function creates the file descriptor which can then be used by subsequent IO functions such as read, write and lseek. The descriptor will around as long as the application is running. In addition to "open" we'll need the equivalent "close" function. Below is an excerpt of the POSIX close manpage from Linux: close(): int close(int fd); close() closes a file descriptor, so that it no longer refers to any file and may be reused. Any record locks (see fcntl(2)) held on the file it was associated with, and owned by the process, are removed (regardless of the file descriptor that was used to obtain the lock). If fd is the last file descriptor referring to the underlying open file description (see open(2)), the resources associated with the open file description are freed; if the descriptor was the last reference to a file which has been removed using unlink(2) the file is deleted. These two functions, open and close, give us the ability to create access to the data for the application at a byte level. Now we need functions that allow us to do byte-range IO to the file. You can probably imagine the first IO functions, which are "read" and "write". Below is an excerpt of the POSIX read function manpage for Linux: ssize_t read(int fd, void *buf, size_t count); read() attempts to read up to count bytes from file descriptor fd into the buffer starting at buf. On files that support seeking, the read operation commences at the current file offset, and the file offset is incremented by the number of bytes read. If the current file offset is at or past the end of file, no bytes are read, and read() returns zero. If count is zero, read() may detect the errors described below. In the absence of any errors, or if read() does not check for errors, a read() with a count of 0 returns zero and has no other effects. If count is greater than SSIZE_MAX, the result is unspecified. In addition to the read() function, there should also be a write() function. Below is an excerpt of the Linux man page for the write() function: write(): ssize_t write(int fd, const void *buf, size_t count); write() writes up to count bytes from the buffer pointed buf to the file referred to by the file descriptor fd. The number of bytes written may be less than count if, for example, there is insufficient space on the underlying physical medium, or the RLIMIT_FSIZE resource limit is encountered (see setrlimit(2)), or the call was interrupted by a signal handler after having written less than count bytes. (See also pipe(7).) For a seekable file (i.e., one to which lseek(2) may be applied, for example, a regular file) writing takes place at the current file offset, and the file offset is incremented by the number of bytes actually written. If the file was open(2)ed with O_APPEND, the file offset is first set to the end of the file before writing. The adjustment of the file offset and the write operation are performed as an atomic step. POSIX requires that a read(2) which can be proved to occur after a write() has returned returns the new data. Note that not all file systems are POSIX conforming. At this point there are open() and close() functions for an object, and there are also read() and write() functions. The last function that is needed to complete basic byte access is lseek(). This IO function is both a curse and a blessing. It can be used to move the file pointer to a new position. The blessing is that it can be very useful for byte-range access in a file and/or deal with complicated data patterns. An excerpt of the man page for lseek for Linux is below: lseek(): off_t lseek(int fd, off_t offset, int whence); The lseek() function repositions the offset of the open file associated with the file descriptor fd to the argument offset according to the directive whence as follows: - SEEK_SET - The offset is set to offset bytes. - SEEK_CUR - The offset is set to its current location plus offset bytes. - SEEK_END - The offset is set to the size of the file plus offset bytes. The lseek() function allows the file offset to be set beyond the end of the file (but this does not change the size of the file). If data is later written at this point, subsequent reads of the data in the gap (a "hole") return null bytes (aq\0aq) until data is actually written into the gap. With these few IO functions, the world of byte-range access to object files is opened. Now you have the option of reading only part of an object instead of having to GET the entire file. This opens a vast new set of possibilities. For example, an application might need to read the first n bytes of a number of files to gather information and present it to the user. With object storage systems, all of the data needs to be first downloaded from the object store to a POSIX file system and read the files. That is a tremendous amount of data movement. With an object-POSIX combination, the files could remain in the object store and the first n bytes read from the file. You can actually put numbers to this example. Imagine having 100 files that are 100MB each. If you need to read the first 16KB of each file, then you are only reading a total of 1.6MB of data. However, if the files are stored in an object store you have to first copy 10 GB worth of data from the object store to a POSIX file system. Then you have to read the 1.6MB of data. That's 1.6MB versus 1001.6MB (a few orders of magnitude difference). This is a wonderful result of adding byte-range functions to object storage — the amount of data either accessed or touched, is much smaller than a pure object storage solution. The Case for Combining Object Storage and POSIX Storage People who use object storage want it to behave more like POSIX storage, but they also want to keep the storage costs at an object level and improve the performance. People who use POSIX file systems like the simplicity of object storage systems and also want the price to come down to object storage levels. In other words, both sides want the rainbow unikitty butterfly. I think it's important to consider what these two groups want. Having an object storage system that allows byte range access is very appealing. It means that rewriting applications to access object storage is now an infinitely easier task. It can also mean that the amount of data touched when reading just a few bytes of a file is greatly reduced (by several orders of magnitude). Think about it. Conceptually the idea has great appeal. Because I'm not a file system developer I can't work out the details, but the end result could be something amazing. Photo courtesy of Shutterstock.