Maybe we need a reminder every now and then that text-files are quite capable and except from being faster to parse in a lot of situation has the benefits of being "future safe" and easy to backup and compress as well.
The common text file for lots of this, CSVs, are absolutely awful. They're fine until they totally aren't, they were just the best option for a lot of use cases. I'd argue that's now been entirely replaced with parquet, significantly faster and broad support, proper types and more.
If you don't already have parquet deployed, there's a wide gulf (in skill set, overhead, etc.) between CSV and parquet.
If you don't mind being old-school, the data is ASCII text, and you're tired of some of CSV's little issue, then ASCII has the FS, GS, RS, and US control characters - specifically intended for such uses. Micro conceptual overhead, none of the CSV issues which screw up many *nix text-handling programs and little script files, and a decent modern filesystem can handle the compression separately.
For many cases of passing data between systems, I'd say the gulf is dramatically lower than it has been until even fairly recently. Support for many standard packages is just there, swapping out "read_csv" for "read_parquet" if you're a pandas shop may even be enough. More and more tools read these directly, and it opens up a whole load of better options for processing data.
> If you don't mind being old-school, the data is ASCII text, and you're tired of some of CSV's little issue, then ASCII has the FS, GS, RS, and US control characters
I have used those before, and yet I still had those characters appear in data. The only places I'd ever seen them were in the wiki page and in customer delivered data. Absolute pain to dig through and remove.
On top of that "if your data is ASCII" is something I'd be nervous about for many use cases even if it is right now.
Beyond that, then you need everyone to swap out their parsing to use those characters.
CSV is fine until it totally blows up in your face. All it takes is one "oh it's fine we'll use awk" stage somewhere or a CSV parser that isn't good enough and one person to put a newline where nobody had expected it before.
> I have used those before, and yet I still had those characters appear in data...
Oh, yes - which is why I emphasized "is". But ASCII text is easy to test for, which lets you fast-track into exception handling - "Tell Sales that Customer data is not as represented", "Trouble-shoot internal data source", etc.
(My experience is that substantial Customer data is never, ever as initially represented. Nor as represented after you point out the first set of issues with it. Nor as represented after you point out the second set of issues. Nor as...)
> Oh, yes - which is why I emphasized "is". But ASCII text is easy to test for, which lets you fast-track into exception handling - "Tell Sales that Customer data is not as represented", "Trouble-shoot internal data source", etc.
Oh with that I mean the ASCII control characters appearing in inputs. So some columns would have record end markers in for example.
If I'm able to make everyone dealing with the reading and writing add specific characters to be used for start/end/etc I'd rather just tell them to swap to a parquet reader unless they've got a really good reason.
As someone in the process of setting up a pipeline and currently using pandas.to_csv as my output, I'm curious what makes you recommend parquet in particular? How does it compare to HDF or Feather?
It depends on what you're trying to optimise, parquet is a very good all round option. HDF I've never really gotten into as it always felt like a good solution only if I move everything over. It's great if your use case fits.
Feather is a layer on top of arrow and was a proof of concept (so I'm not sure how heavily it's used now), and arrow is fast becoming the interchange format. It's exactly laid out as things will be in memory - which means zero copy for shuttling it around from one place to another. I _think_ there is less support for feather but that is likely changing as everything converges.
Parquet should be
* Faster to write
* Faster to read (even if you're reading the whole file, which actually isn't required, the format helps you read just sections of the columns you need)
* Smaller
* Better at handling actual floating points
than CSV, while having actual standards alongside it. Be a little wary of pandas guessing the right column types for you if you're creating partitioned files btw.
When you're working with pandas, etc (check out Dask) you can pretty much just swap out some reading and writing functions. You can also use pyarrow directly if you need to be very careful about column types.
For your use case you may want to explicitly use a single column for the features that is a list, I'm not sure if that's better/worse than having so many columns. If a reader may want to find just some images where a small subset of features are > X, you might benefit from multiple columns so that the reader only processes the data it needs.
Worth testing out, but I expect you should be able to try it out in an afternoon if you're already working with pandas/similar. Just install pyarrow and use a to_parquet. Things like dask (or straight pyarrow) give you partitioned files as output if you want too, if there's a useful column or columns to split on https://arrow.apache.org/docs/python/parquet.html#partitione...
Just a heads up, pandas to Csv is laughably slow. You're literally better off converting the pandas df to arrow and then serialise to Csv with arrow, though the format is quite limited with arrow.
Well at least it’s better than numpy’s csv functions. For my use case (~4 million images, one row per image, 2048 fp32 columns for features extracted with resnet), parsing/encoding csv with pandas was 10x faster than with numpy. Of course serializing the arrays to/from npy was 10x faster than any csv library.
Yes - parquet tooling is non-existent compared to CSV particularly on the command line. And the cross-language/platform support is a mess - good luck reading Pandas generated parquet on .NET or in a (non-spark) JVM environment.
There are many reasons why CSV is flawed for the purposes of storing tabular data (e.g. loss of column type information) but the alternatives are just so unergonomic that CSV remains a viable choice in many situations.
Column-oriented formats such as Parquet can be awful too. For example, if you have five columns of numbers you want to multiply together, Parquet is going to be the worst option available because of the extreme cache thrashing that the CPU will encounter.
Structure packing[1] and consideration of locality of reference[2] would need to be applied for high performance applications where a computer scientist has considered the algorithm needing to be implemented and the most efficient data format that the source data would need to be provided in.
Exactly. Basically most big data lives in hdfs and hdfs is part of hadoop. Even if you use Spark and Flink I would classify that as using Hadoop (under the hood).
Aside from Apache Spark, what's replaced it and does it still face the same speed of access limitations compared to just zipping through giant CSVs with awk or whatever streaming APIs you write with your own preferred language?
I use Spark for a number of jobs for language-specific features still but I think within 2 years all custom code will be trivially invoked as native UDFs in SQL data warehouses (ie Snowflake, which has essentially solved big-data performance as a going concern).
I just write SQL in Snowflake and it replaces 95% of what I would otherwise have done in custom MapReduce or Spark code.
What I really dislike modern cloud DWH such as Snowflake is that it hides a lot of things from me. Since I'm not a CTO who worries about not delivering, but a junior DE who actually wants to learn things, I really prefer that things were done in the old ways where we had to manage our own infrastructure and our own code for ETL. These kinds of things can not be learned "just for fun" because one has to work in a real environment.
What do you mean? Of course you can still learn them "just for fun" if you want. There are plenty of columnar data warehouses (memsql, greenplum, vertica, clickhouse, etc) and data processing frameworks (spark, flink, etc) that you can look at, implement and run yourself.
What I'm saying is that you can surely scratch the basics from personal use, but it's completely different from real usage and such can only be trained on job. Now that those jobs are fewer as everyone goes on cloud.
SQL will always be faster than Hadoop and MapReduce. The main reason to use those other slower services is developer are not use to SQL or declarative programming, and insist on having the code in Procedural way.
That's completely backwards. Mapreduce-like approaches are how SQL datastores are implemented underneath; the absolute best case for SQL is to equal hand-tuned mapreduce-like performance, and often it will be slower (you're at the mercy of your query planner to pick the right indices, do joins in the right order, etc.). The main reason people use SQL is because they find it easier to express a query that way (which is completely legitimate - if your query planner is good enough most of the time, you've got better things to be doing than hand-tuning your query execution).
No, that does not seem correct. SQL Datastores are not "map-reduce underneath", they have optimized datastructures for efficient querying (i.e. indices). Map-reduce is equivalent to those cases in SQL database where you have full table scan in your query plan - basically brute-forcing your way through the dataset.
You can (and often should) have indices in a map-reduce situation as well - you just build them in an explicit, visible way. But in most of the relevant use cases you're doing some kind of aggregation over the whole table, so indices don't help any.
And if your primary use-case is column-wise aggregation over the whole table, in SQL you'd use a (compressed) column store rather than a row store as your table storage method.
To be fair, Parquet, which is commonly used in Big Data solutions is a column store format. So, once you normalize your data and save it as Parquet you can have efficient column-wise aggregation - but that assumes some preprocessing step.
That makes no sense. SQL is a query language, commonly implemented by relational databases.
In the early 2000s, columnar relational data warehouses were not sophisticated and scalable enough to handle the scale of data encountered at Yahoo, Google and other internet companies. MapReduce (and the many evolutions of Hadoop ecosystem) was created to scale processing through low-level instructions and algorithms.
Eventually columnar data warehouses caught up and are now capable of handling petabyte scale, regardless of whatever language you use to query them. The fundamental storage and compute primitives haven't really changed that much, just offered in a much more user-friendly way now.
SQL itself is just a query language, it's the underlying cloud based data warehouse that fulfills the role of what map/reduce used to do in terms of parallelization transparently.
1) Why would you want to maintain your own Spark infrastructure? Spark on Kube is a huge improvement over YARN but you still have to deal with OOMEs, filled disks, Kube upgrades, pushing custom images to container registries, etc etc etc.
2) Snowflake is probably 10-50x as performant as Spark for data manipulation. I don't know what kind of unholy demonic incantations Snowflake is doing on the backend to support their SQL performance, but it's really freaking fast. There's just no other way to cut it.
I've spent 5-10 years eking every ounce of performance I can get out of a Hadoop/Spark cluster. I'm not trying to be unreasonable about this. I would love for OSS to be competitive; it's great for the world, and it would be great for my skill set and earning potential.
But it's not a contest, and if you think standalone Spark is going to be a viable competitor in a couple years, you are deluding yourself. Make informed choices about your career and investment.
A lot of articles I read about snowflake involves data vault which is a massive turn off. And when their tech lead (Kent Graziano) is a prominent figure in the DV bullshit...
Snowflake and DV have no interdependency whatsoever. Snowflake is just a database. Whether you use DV to model the data inside of it or dimensional modelling or "big wide tables" is completely up to you, there's nothing about it that requires or benefits DV in particular.
You should try Databricks, especially the new Photon engine powering Spark. In general more performant than Snowflake in SQL and a lot more flexible. (There are some cases in which Databricks would be slower but the perf is improving rapidly.)
Databricks has an extremely bad API. So, sure, your Spark jobs might be a little bit faster some times, but why would you use it if you can't even read logs of running jobs?
Databricks is amazing, the Delta Live Table technology is incredible. It's very hard to approach problems like Data Lineage and Data Quality, but that platform does it in the right way.
My only concern is that they offer just a managed cloud product. That's cool for startups, but large enterprises sometimes need more governance and ownership than that.
I spent a while putting a POC of Flink for a use case together at client behest a couple of years ago. Really struggled with it. Seemed slow as heck too. I'd much rather write a few hundred extra lines of my own code to do stuff like that (ingest -> process -> output. Modern software fashion trends seems to create as much work as it tries to save; as extensive and magical as a lot of the features are.
Whether a technology can replace Hadoop in an organization depends on many factors, but some technologies that solve at least in part similar problem are Apache Storm, Spark, Flink, Kafka Streams,
and maybe BigQuery?
Or, as the original article says, some companies just use some command line tools, shell scripts.
It's been a couple of years since I was interested in Data Engineering, so my knowledge on this topic is some years behind.
I've not seen Storm being used anywhere sane for a few years at least now, and from a glance at job postings it looks unlikely. Spark, Kafka Streams etc. are definitely used in a modern data platform from my experience.
I think we're seeing a big shift with Hadoop-like workloads being moved onto cloud providers, so BigQuery, Amazon EMR etc.
I'm curious what constitutes "big data" anymore. In an intermediate machine learning course, we train on nearly a petabyte of data using Google Colab and Jupyter Notebooks. Nobody discusses the size of the data requiring any special treatment due to its size... would not 95% of a petabyte be "big data"?
What course are you taking? Imagenet is only 150 GB, and Common Crawl is only 320 TB.
Big data is a moving target, but I’m comfortable defining it as data too large to fit in memory. Obviously, you can always get a bigger node, my rule is thumb is that if you need generators, you are working with big data.
Glue often uses EMR under the hood, which is often Spark. And Athena is PrestoDB, as far as I know it has nothing to do with Hadoop other than you can use it to query Hadoop data stores.
The way I see it, Hadoop is still in common use as the storage layer for Spark and related implementations, whether that is in the form of HDFS or something like EMRFS:
Quote from AWS: "EMRFS is an implementation of the Hadoop file system ..."
Ah - that's a good point. Usually when people say "Hadoop", I assume they're referring to HDFS, but there is (was...) the Hadoop MR system that ran on top of it that has been almost entirely replaced by Spark.
> What about if you had 500 petabytes of data to analyze?
Then we probably need the big tools.
> Big data does exist in the wild.
So does little data.
The problem is that a "one-size-fits-all" approach has become common, not just in data analysis; think of all the low-medium traffic webpages that use giant, complex frameworks and huge distributed systems just to display essentially a small CRUD app that would have been ALOT easier to cobble together in plain JS on a simple LAMP server.
What the article shows is the importance on deciding for the right tool for the job: When I want to plant a little tree in my backyard, bringing one of these
https://upload.wikimedia.org/wikipedia/commons/0/01/Bucket_w...
to dig the hole is proooobably overengineering it a tiny little bit, and will likely take longer than getting a shovel.
Those 500 PB imply a competent staff, a world-class infrastructure, and worthwhile applications. These things don't materialize suddenly: they start maturing when the data set is 500 GB or even just planned and they usually mature enough to switch technology multiple times as the scale rises.
The article still applies to the things which have replaced Hadoop. The operational, development, and computational overhead of distributing/re-aggregating work remains huge.
Outside of legacy systems, Hadoop isn't widely used anymore.