{"id":537,"date":"2020-10-06T07:45:20","date_gmt":"2020-10-06T07:45:20","guid":{"rendered":"http:\/\/feellikelearning.com\/?p=537"},"modified":"2020-11-27T09:03:05","modified_gmt":"2020-11-27T09:03:05","slug":"cn-how-to-read-orc-file-and-convert-it-to-a-pandas-dataframe","status":"publish","type":"post","link":"https:\/\/feellikelearning.com\/index.php\/2020\/10\/06\/cn-how-to-read-orc-file-and-convert-it-to-a-pandas-dataframe\/","title":{"rendered":"\u600e\u6837\u628aorc\u6587\u4ef6\u8bfb\u8fdb\u6765\u53d8\u6210pandas dataframe ? | how to read orc file and convert it to a pandas dataframe ?"},"content":{"rendered":"\n<p>\u7406\u8bba\u4e0a\u6765\u8bf4\uff0cpandas\u63d0\u4f9b\u4e86\u4e2a\u5de5\u5177<\/p>\n\n\n\n<!--more-->\n\n\n\n<pre class=\"wp-block-code\"><code>import pandas as pd\n\ndf = pd.read_orc(file_name)<\/code><\/pre>\n\n\n\n<p>\u7136\u800c\uff0cpandas\u5e95\u5c42\u7528\u4e86\u4e2a\u53ebpyarrow\u7684library\uff0c\u7136\u540epip install \u91cc\u9762\u7684image\u628apyarrow._orc\u90e8\u5206\u62ff\u6389\u4e86\uff0c\u4ed6\u4eec\u6709\u4e9borc\u7684C++ link error\u3002<\/p>\n\n\n\n<p>\u8fd9\u4e2aYoutube\u89c6\u9891\u89e3\u91ca\u7684\u5f88\u6e05\u695a\u5e76\u4e14\u7ed9\u51fa\u4e86\u89e3\u51b3\u4e4b\u9053\uff0c\u5c31\u662f\u7528\u53e6\u4e00libary pyorc<\/p>\n\n\n\n<figure class=\"wp-block-embed-youtube wp-block-embed is-type-video is-provider-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio\"><div class=\"wp-block-embed__wrapper\">\n<iframe loading=\"lazy\" title=\"How I solved &quot;No module named &#039;pyarrow._orc&#039; error? - ORC to dataframe in Python\" width=\"1130\" height=\"636\" src=\"https:\/\/www.youtube.com\/embed\/qvV_Frc6zB8?start=1&#038;feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen title=\"How I solved \"No module named 'pyarrow._orc' error? - ORC to dataframe in Python\"><\/iframe>\n<\/div><\/figure>\n\n\n\n<p>\u5177\u4f53error\u4e3a<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>import pyarrow._orc<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>---------------------------------------------------------------------------\nModuleNotFoundError                       Traceback (most recent call last)\n&lt;ipython-input-65-ca95b0424328> in &lt;module>\n----> 1 import pyarrow._orc\n\nModuleNotFoundError: No module named 'pyarrow._orc'\n<\/code><\/pre>\n\n\n\n<p>google\u4e86\u4e00\u4e0b\uff0c\u8c8c\u4f3cmac\u548cwindows\u90fd\u6709\u8fd9\u4e2a\u95ee\u9898\u3002\u4f30\u8ba1\u4e00\u65f6\u534a\u4f1a\u4e5f\u4fee\u4e0d\u597d\uff0c\u4ed6\u4eec\u7684ticket\u8bf4\u627e\u4e0d\u5230\u81ea\u613f\u8005\u3002\u6211\u8bd5\u8fc7\u53e6\u4e00\u79cd\u5e38\u7528\u683c\u5f0fparquet\u5012\u662f\u597d\u7684\u3002<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>df = pd.read_parquet(file_name)<\/code><\/pre>\n\n\n\n<p>\u8fd9\u91cc\u6211\u5199\u4e86\u4e2a\u53ef\u4ee5\u76f4\u63a5\u7528\u7684function\uff0c\u8f93\u5165\u4e3aorc file directory\uff0c\u7136\u540e\u8bfb\u5165\u76ee\u5f55\u4e0b\u5168\u90e8\u6587\u4ef6\u5e76\u5408\u5e76\u6210\u4e00\u4e2apandas dataframe\u5e76\u8fd4\u56de\u3002\u4e3a\u4ec0\u4e48\u8981\u8bfb\u5165\u591a\u4e2a\u6587\u4ef6\uff1f\u56e0\u4e3a\u5728\u5927\u6570\u636e\u5904\u7406\u7cfb\u7edf\u91cc\u9762\uff0c\u6bd4\u5982Hive\u6216Spark\uff0c\u6570\u636e\uff08\u6bd4\u5982\uff0cHive table)\u90fd\u662f\u5206\u6210\u5c0f\u6587\u4ef6\u7684\uff08part files\uff09\u3002ptn\u662fpattern\uff0c\u6709\u65f6\u4f60\u60f3\u8df3\u8fc7\u4e00\u4e9b\u975e\u6570\u636e\u6587\u4ef6\uff0c\u6bd4\u5982_SUCCESS file\u4e4b\u7c7b\u3002<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>def orc_to_df(path, ptn='**\/*'):\n    dfs = []\n    \n    paths = list(Path(path).glob(ptn))\n    print(\"to read files: \")\n    for path in paths:\n        print(path)\n    \n    i = 0\n    for path in paths:\n        f = open(path, 'rb')\n        reader = pyorc.Reader(f) \n        columns = reader.schema.fields\n\n        # sort by column id to ensure correct order (since \"fields\" is a dict, order may not be correct)\n        columns = [y for x, y in sorted([(reader.schema.find_column_id(c), c) for c in columns])] \n        df = pd.DataFrame(reader, columns=columns)\n        dfs += [df]\n        i += 1\n        print(f'loaded {i} part file(s).')\n        \n    df = pd.concat(dfs)\n    return df<\/code><\/pre>\n\n\n\n<p><\/p>\n","protected":false},"excerpt":{"rendered":"<p>\u7406\u8bba\u4e0a\u6765\u8bf4\uff0cpandas\u63d0\u4f9b\u4e86\u4e2a\u5de5\u5177<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_mi_skip_tracking":false},"categories":[1],"tags":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v19.10 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>\u600e\u6837\u628aorc\u6587\u4ef6\u8bfb\u8fdb\u6765\u53d8\u6210pandas dataframe ? | how to read orc file and convert it to a pandas dataframe ? | Feel Like Learning<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/feellikelearning.com\/index.php\/2020\/10\/06\/cn-how-to-read-orc-file-and-convert-it-to-a-pandas-dataframe\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"\u600e\u6837\u628aorc\u6587\u4ef6\u8bfb\u8fdb\u6765\u53d8\u6210pandas dataframe ? | how to read orc file and convert it to a pandas dataframe ? | Feel Like Learning\" \/>\n<meta property=\"og:description\" content=\"\u7406\u8bba\u4e0a\u6765\u8bf4\uff0cpandas\u63d0\u4f9b\u4e86\u4e2a\u5de5\u5177\" \/>\n<meta property=\"og:url\" content=\"https:\/\/feellikelearning.com\/index.php\/2020\/10\/06\/cn-how-to-read-orc-file-and-convert-it-to-a-pandas-dataframe\/\" \/>\n<meta property=\"og:site_name\" content=\"Feel Like Learning\" \/>\n<meta property=\"article:published_time\" content=\"2020-10-06T07:45:20+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2020-11-27T09:03:05+00:00\" \/>\n<meta name=\"author\" content=\"feellikelearning\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"feellikelearning\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"1 minute\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/feellikelearning.com\/index.php\/2020\/10\/06\/cn-how-to-read-orc-file-and-convert-it-to-a-pandas-dataframe\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/feellikelearning.com\/index.php\/2020\/10\/06\/cn-how-to-read-orc-file-and-convert-it-to-a-pandas-dataframe\/\"},\"author\":{\"name\":\"feellikelearning\",\"@id\":\"https:\/\/feellikelearning.com\/#\/schema\/person\/91fb815bebebf166c217b5e3764d437a\"},\"headline\":\"\u600e\u6837\u628aorc\u6587\u4ef6\u8bfb\u8fdb\u6765\u53d8\u6210pandas dataframe ? | how to read orc file and convert it to a pandas dataframe ?\",\"datePublished\":\"2020-10-06T07:45:20+00:00\",\"dateModified\":\"2020-11-27T09:03:05+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/feellikelearning.com\/index.php\/2020\/10\/06\/cn-how-to-read-orc-file-and-convert-it-to-a-pandas-dataframe\/\"},\"wordCount\":65,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/feellikelearning.com\/#\/schema\/person\/91fb815bebebf166c217b5e3764d437a\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/feellikelearning.com\/index.php\/2020\/10\/06\/cn-how-to-read-orc-file-and-convert-it-to-a-pandas-dataframe\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/feellikelearning.com\/index.php\/2020\/10\/06\/cn-how-to-read-orc-file-and-convert-it-to-a-pandas-dataframe\/\",\"url\":\"https:\/\/feellikelearning.com\/index.php\/2020\/10\/06\/cn-how-to-read-orc-file-and-convert-it-to-a-pandas-dataframe\/\",\"name\":\"\u600e\u6837\u628aorc\u6587\u4ef6\u8bfb\u8fdb\u6765\u53d8\u6210pandas dataframe ? | how to read orc file and convert it to a pandas dataframe ? | Feel Like Learning\",\"isPartOf\":{\"@id\":\"https:\/\/feellikelearning.com\/#website\"},\"datePublished\":\"2020-10-06T07:45:20+00:00\",\"dateModified\":\"2020-11-27T09:03:05+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/feellikelearning.com\/index.php\/2020\/10\/06\/cn-how-to-read-orc-file-and-convert-it-to-a-pandas-dataframe\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/feellikelearning.com\/index.php\/2020\/10\/06\/cn-how-to-read-orc-file-and-convert-it-to-a-pandas-dataframe\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/feellikelearning.com\/index.php\/2020\/10\/06\/cn-how-to-read-orc-file-and-convert-it-to-a-pandas-dataframe\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/feellikelearning.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"\u600e\u6837\u628aorc\u6587\u4ef6\u8bfb\u8fdb\u6765\u53d8\u6210pandas dataframe ? | how to read orc file and convert it to a pandas dataframe ?\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/feellikelearning.com\/#website\",\"url\":\"https:\/\/feellikelearning.com\/\",\"name\":\"Feel Like Learning\",\"description\":\"\u7a0b\u5e8f\uff5c\u751f\u6d3b\uff5c\u5b66\u5230\u5c31\u662f\u8d5a\u5230\",\"publisher\":{\"@id\":\"https:\/\/feellikelearning.com\/#\/schema\/person\/91fb815bebebf166c217b5e3764d437a\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/feellikelearning.com\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":[\"Person\",\"Organization\"],\"@id\":\"https:\/\/feellikelearning.com\/#\/schema\/person\/91fb815bebebf166c217b5e3764d437a\",\"name\":\"feellikelearning\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/feellikelearning.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/72a1e86e9dcb0332e88bd7d54fd36c28?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/72a1e86e9dcb0332e88bd7d54fd36c28?s=96&d=mm&r=g\",\"caption\":\"feellikelearning\"},\"logo\":{\"@id\":\"https:\/\/feellikelearning.com\/#\/schema\/person\/image\/\"},\"url\":\"https:\/\/feellikelearning.com\/index.php\/author\/feellikelearning\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"\u600e\u6837\u628aorc\u6587\u4ef6\u8bfb\u8fdb\u6765\u53d8\u6210pandas dataframe ? | how to read orc file and convert it to a pandas dataframe ? | Feel Like Learning","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/feellikelearning.com\/index.php\/2020\/10\/06\/cn-how-to-read-orc-file-and-convert-it-to-a-pandas-dataframe\/","og_locale":"en_US","og_type":"article","og_title":"\u600e\u6837\u628aorc\u6587\u4ef6\u8bfb\u8fdb\u6765\u53d8\u6210pandas dataframe ? | how to read orc file and convert it to a pandas dataframe ? | Feel Like Learning","og_description":"\u7406\u8bba\u4e0a\u6765\u8bf4\uff0cpandas\u63d0\u4f9b\u4e86\u4e2a\u5de5\u5177","og_url":"https:\/\/feellikelearning.com\/index.php\/2020\/10\/06\/cn-how-to-read-orc-file-and-convert-it-to-a-pandas-dataframe\/","og_site_name":"Feel Like Learning","article_published_time":"2020-10-06T07:45:20+00:00","article_modified_time":"2020-11-27T09:03:05+00:00","author":"feellikelearning","twitter_card":"summary_large_image","twitter_misc":{"Written by":"feellikelearning","Est. reading time":"1 minute"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/feellikelearning.com\/index.php\/2020\/10\/06\/cn-how-to-read-orc-file-and-convert-it-to-a-pandas-dataframe\/#article","isPartOf":{"@id":"https:\/\/feellikelearning.com\/index.php\/2020\/10\/06\/cn-how-to-read-orc-file-and-convert-it-to-a-pandas-dataframe\/"},"author":{"name":"feellikelearning","@id":"https:\/\/feellikelearning.com\/#\/schema\/person\/91fb815bebebf166c217b5e3764d437a"},"headline":"\u600e\u6837\u628aorc\u6587\u4ef6\u8bfb\u8fdb\u6765\u53d8\u6210pandas dataframe ? | how to read orc file and convert it to a pandas dataframe ?","datePublished":"2020-10-06T07:45:20+00:00","dateModified":"2020-11-27T09:03:05+00:00","mainEntityOfPage":{"@id":"https:\/\/feellikelearning.com\/index.php\/2020\/10\/06\/cn-how-to-read-orc-file-and-convert-it-to-a-pandas-dataframe\/"},"wordCount":65,"commentCount":0,"publisher":{"@id":"https:\/\/feellikelearning.com\/#\/schema\/person\/91fb815bebebf166c217b5e3764d437a"},"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/feellikelearning.com\/index.php\/2020\/10\/06\/cn-how-to-read-orc-file-and-convert-it-to-a-pandas-dataframe\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/feellikelearning.com\/index.php\/2020\/10\/06\/cn-how-to-read-orc-file-and-convert-it-to-a-pandas-dataframe\/","url":"https:\/\/feellikelearning.com\/index.php\/2020\/10\/06\/cn-how-to-read-orc-file-and-convert-it-to-a-pandas-dataframe\/","name":"\u600e\u6837\u628aorc\u6587\u4ef6\u8bfb\u8fdb\u6765\u53d8\u6210pandas dataframe ? | how to read orc file and convert it to a pandas dataframe ? | Feel Like Learning","isPartOf":{"@id":"https:\/\/feellikelearning.com\/#website"},"datePublished":"2020-10-06T07:45:20+00:00","dateModified":"2020-11-27T09:03:05+00:00","breadcrumb":{"@id":"https:\/\/feellikelearning.com\/index.php\/2020\/10\/06\/cn-how-to-read-orc-file-and-convert-it-to-a-pandas-dataframe\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/feellikelearning.com\/index.php\/2020\/10\/06\/cn-how-to-read-orc-file-and-convert-it-to-a-pandas-dataframe\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/feellikelearning.com\/index.php\/2020\/10\/06\/cn-how-to-read-orc-file-and-convert-it-to-a-pandas-dataframe\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/feellikelearning.com\/"},{"@type":"ListItem","position":2,"name":"\u600e\u6837\u628aorc\u6587\u4ef6\u8bfb\u8fdb\u6765\u53d8\u6210pandas dataframe ? | how to read orc file and convert it to a pandas dataframe ?"}]},{"@type":"WebSite","@id":"https:\/\/feellikelearning.com\/#website","url":"https:\/\/feellikelearning.com\/","name":"Feel Like Learning","description":"\u7a0b\u5e8f\uff5c\u751f\u6d3b\uff5c\u5b66\u5230\u5c31\u662f\u8d5a\u5230","publisher":{"@id":"https:\/\/feellikelearning.com\/#\/schema\/person\/91fb815bebebf166c217b5e3764d437a"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/feellikelearning.com\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":["Person","Organization"],"@id":"https:\/\/feellikelearning.com\/#\/schema\/person\/91fb815bebebf166c217b5e3764d437a","name":"feellikelearning","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/feellikelearning.com\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/72a1e86e9dcb0332e88bd7d54fd36c28?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/72a1e86e9dcb0332e88bd7d54fd36c28?s=96&d=mm&r=g","caption":"feellikelearning"},"logo":{"@id":"https:\/\/feellikelearning.com\/#\/schema\/person\/image\/"},"url":"https:\/\/feellikelearning.com\/index.php\/author\/feellikelearning\/"}]}},"_links":{"self":[{"href":"https:\/\/feellikelearning.com\/index.php\/wp-json\/wp\/v2\/posts\/537"}],"collection":[{"href":"https:\/\/feellikelearning.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/feellikelearning.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/feellikelearning.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/feellikelearning.com\/index.php\/wp-json\/wp\/v2\/comments?post=537"}],"version-history":[{"count":5,"href":"https:\/\/feellikelearning.com\/index.php\/wp-json\/wp\/v2\/posts\/537\/revisions"}],"predecessor-version":[{"id":762,"href":"https:\/\/feellikelearning.com\/index.php\/wp-json\/wp\/v2\/posts\/537\/revisions\/762"}],"wp:attachment":[{"href":"https:\/\/feellikelearning.com\/index.php\/wp-json\/wp\/v2\/media?parent=537"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/feellikelearning.com\/index.php\/wp-json\/wp\/v2\/categories?post=537"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/feellikelearning.com\/index.php\/wp-json\/wp\/v2\/tags?post=537"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}