<p>I think this is going to be one of those impossible to answer questions because of so many unknown variables.</p>

<p>MySQL could be screwing it up because it depends on 1) MySQL server configuration and 2) The charset and collation of the database, the tables, the connection strings, etc.</p>

<p>The file system itself could be screwing it up because they all tend to have different treatments for character encodings (HFS, NTFS, FAT, EXT, etc.).</p>

<p>Throw in the fact that you said this has been on various developer machines and touched by an unknown number of other programs, it makes it very difficult to diagnose.</p>

CC BY-SA 3.0
2015-04-16T18:57:13.417Z
Given this, and I've requested this several times both for the EE and Craft versions of Craft, can we please, please, please have a setting to convert  filenames to ascii, like slugs? :)
André Elvan
CC BY-SA 3.0
2015-04-16T19:02:10.723Z
@AndréElvan Already on our internal list. :)
Brad Bell
CC BY-SA 3.0
2015-04-17T08:36:24.450Z
Brad: I totally understand :) I just wanted to collect experiences from other people!
gromgull
CC BY-SA 3.0
2015-04-17T08:38:15.253Z
I also take this to mean that "Craft does not intentionally apply unicode normalisation to filenames (or to anything)" - so it happened "somewhere else".
gromgull
CC BY-SA 3.0
2015-04-17T15:57:16.063Z
@gromgull Correct... Craft isn't doing any filename normalization currently.
Brad Bell
CC BY-SA 3.0
2015-04-16T17:44:01.687Z
2015-04-16T17:44:01.687Z
Brad Bell
<p>This has now been solved by the addition of the <a href="https://craftcms.com/docs/config-settings#convertFilenamesToAscii" rel="nofollow"><code>convertFilenamesToAscii</code></a> config setting introduced in Craft 2.4.</p>

<p>If you are working on a multilingual site, you might also find the <a href="https://craftcms.com/docs/config-settings#limitAutoSlugsToAscii" rel="nofollow"><code>limitAutoSlugsToAscii</code></a> config setting useful.</p>

CC BY-SA 3.0
2016-06-08T19:22:37.643Z
2016-06-08T19:22:37.643Z
Steve Rowling
<p>We have a site deployed in Norway, and people have uploaded PDF-assets with Norwegian non-ascii characters in the filenames. </p>

<p>Normally this works fine, the URLs are encoded like this and it works: </p>

<p><a href="http://[oursite].no/assets/files/downloads/%C3%85rsberetning-2014_kladd.pdf" rel="nofollow">http://[oursite].no/assets/files/downloads/%C3%85rsberetning-2014_kladd.pdf</a></p>

<p>("Årsberetning" - annual report)</p>

<p>But one file didn't want to download, it only gave me 404. However, checking on the server the file was there, and the filename appeared correct. Also, re-scanning asset folders didn't remove it. </p>

<p>BUT - I was not able to rename the file from within craft, nor download it. </p>

<p>After much hair-pulling I established that the file on disk had the Å encoded as '\x41\xcc\x8a' whereas craft looked for a "pre-composed" Å ('\xc3\x85'). </p>

<p>NOW the question - where did this break? :) </p>

<p>Does Craft unicode-normalise filenames? 
Does MySQL normalise the unicode it stores? </p>

<p>The file has moved around various developer machines and staging servers before making it into production, it could be some other part of the tool-chain that broke it. </p>

CC BY-SA 3.0
2015-04-16T18:55:51.750Z
I'm having problems with this all the time with norwegian characters. For me, the problem normally happends when moving files between mac osx and various linux servers. As long as the files is uploaded to craft to a server, and stay on that server, it works fine.
André Elvan
CC BY-SA 3.0
2015-04-17T08:38:55.843Z
André - yeah, that is my experience as well. On our linux staging server the file was uploaded, there it worked fine (but the file was stored twice, with both encodings...) - then it went through git onto a osx machine, and from there it was pushed live with rsync. It's actually a miracle that anything ever works at all :)
gromgull
<assets><filesystem>
Asset filename unicode normalisation?
# Asset filename unicode normalisation?

(Asked by: username_0 on 2015-04-16)

We have a site deployed in Norway, and people have uploaded PDF-assets with Norwegian non-ascii characters in the filenames.

Normally this works fine, the URLs are encoded like this and it works:

<http://[oursite].no/assets/files/downloads/%C3%85rsberetning-2014_kladd.pdf>

("Årsberetning" - annual report)

But one file didn't want to download, it only gave me 404. However, checking on the server the file was there, and the filename appeared correct. Also, re-scanning asset folders didn't remove it.

BUT - I was not able to rename the file from within craft, nor download it.

After much hair-pulling I established that the file on disk had the Å encoded as '\x41\xcc\x8a' whereas craft looked for a "pre-composed" Å ('\xc3\x85').

NOW the question - where did this break? :)

Does Craft unicode-normalise filenames?
Does MySQL normalise the unicode it stores?

The file has moved around various developer machines and staging servers before making it into production, it could be some other part of the tool-chain that broke it.

**Tags:** assets, filesystem
> **Comments:**
> **username_1**: I'm having problems with this all the time with norwegian characters. For me, the problem normally happends when moving files between mac osx and various linux servers. As long as the files is uploaded to craft to a server, and stay on that server, it works fine.  
> **username_0**: André - yeah, that is my experience as well. On our linux staging server the file was uploaded, there it worked fine (but the file was stored twice, with both encodings...) - then it went through git onto a osx machine, and from there it was pushed live with rsync. It's actually a miracle that anything ever works at all :)  

---

## 2 answers

### Answer by username_2 on 2015-04-16. Score: 3 | Accepted answer

I think this is going to be one of those impossible to answer questions because of so many unknown variables.

MySQL could be screwing it up because it depends on 1) MySQL server configuration and 2) The charset and collation of the database, the tables, the connection strings, etc.

The file system itself could be screwing it up because they all tend to have different treatments for character encodings (HFS, NTFS, FAT, EXT, etc.).

Throw in the fact that you said this has been on various developer machines and touched by an unknown number of other programs, it makes it very difficult to diagnose.

> **Comments:**
> **username_1**: Given this, and I've requested this several times both for the EE and Craft versions of Craft, can we please, please, please have a setting to convert  filenames to ascii, like slugs? :)  
> **username_2**: @AndréElvan Already on our internal list. :)  
> **username_0**: Brad: I totally understand :) I just wanted to collect experiences from other people!  
> **username_0**: I also take this to mean that "Craft does not intentionally apply unicode normalisation to filenames (or to anything)" - so it happened "somewhere else".  
> **username_2**: @gromgull Correct... Craft isn't doing any filename normalization currently.  

---

### Answer by username_3 on 2016-06-08. Score: 4

This has now been solved by the addition of the [`convertFilenamesToAscii`](https://craftcms.com/docs/config-settings#convertFilenamesToAscii) config setting introduced in Craft 2.4.

If you are working on a multilingual site, you might also find the [`limitAutoSlugsToAscii`](https://craftcms.com/docs/config-settings#limitAutoSlugsToAscii) config setting useful.


---

