If your data are rectangular and you care about performance, CSV is better than JSON just because it avoids repetitive key names everywhere. Then again, if your data are rectangular and you really care about performance, you would not use any of these (you might use HDF5, which has support in many programming languages and will destroy the others in terms of speed).
JSON is almost a subset of CSV, with the understanding that you have to wrap every line in [], the document in [], and every field must be quoted. (And JSON doesn't have built-in support for headings, so you need to write a little loop instead of the library building a hash for you.)
So no, if you control input and output, JSON is still easier to use than CSV, and just as performant. JSON stores straight arrays just fine. It's not the format's fault so many people choose to store hashes with it.
...and quotes inside fields must be escaped with a backslash, newlines replaced by \n, etc.
...and, the moment you declare you handle json, people will send non-string data ("that is a number, of course it isn't quoted"), attempt to include nested data, leave out the opening and closing [] (because people will grep a file with one array per line to filter a json file; that is no way robust, but people will do it, anyways)
The main advantage of json vs csv is that there is only one json, but quite a few different variant of csv. That is a huge advantage, but csv is engrained; there are many, many tools that handle csv but not json.
And the "if you control input and output" case is not the interesting or the problematic one.
