Piping, splitting, and zipping a CSV in LinuxWhat is the best way to find a list of several strings within a...
What is the meaning of "log" in this sentence?
How acceptable is an ellipsis "..." in formal mathematics?
How to ride a fish?
How to make "acts of patience" exciting?
Creating chess engine, machine learning vs. traditional engine?
Is it possible to use a Mosfet in switching mode as an electronic load?
What determines the top speed in ice skating?
Are the names of guilds in Ravnica copyrighted?
What does IKEA-like mean?
均等 is noun or adjective
How do lasers measure short distances (<1cm) when electronics are too slow for time-of-flight to work?
Is it fine to ask this kind of question to the corresponding author of a paper?
Why did a young George Washington sign a document admitting to assassinating a French military officer?
Does journal access significantly influence choice in which journal to publish in?
Relation between signal processing and control systems engineering?
Does the Creighton Method of Natural Family Planning have a failure rate of 3.2% or less?
Can a Creature at 0 HP Take Damage?
My first random password generator
Overlay image with parts of another image
D&D Monsters and Copyright
Should I withdraw my paper because the Editor is behaving so badly with me?
Would my post-apocalyptic US Government be able to work and function properly?
Finger Picking Chords - Beats per bar
How to temporarily replace Latin characters with Greek
Piping, splitting, and zipping a CSV in Linux
What is the best way to find a list of several strings within a large text fileIssues Splitting CSV filesawk and system and print and append to csv fileCount CSV columns and removeUsing bash to swap first and second columns in CSVSplitting text file into CSV with multiple delimiters in bash?csv filter on linux bash
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{
margin-bottom:0;
}
I have some large CSV that need to be processed. I need them pipe delimited, split into about 900,000 rows each, and gzipped
I have tried several methods. Currently I am using pandas to process the CSVs, everything else is written in Python. It just feels a little too slow.
What’s the fastest method?
shell-script awk python csv csvkit
asked 51 mins ago
Jedi_mcJedi_mc
1
New contributor
Jedi_mc is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
add a comment
|
I have some large CSV that need to be processed. I need them pipe delimited, split into about 900,000 rows each, and gzipped
I have tried several methods. Currently I am using pandas to process the CSVs, everything else is written in Python. It just feels a little too slow.
What’s the fastest method?
shell-script awk python csv csvkit
asked 51 mins ago
Jedi_mcJedi_mc
1
New contributor
Jedi_mc is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
add a comment
|
I have some large CSV that need to be processed. I need them pipe delimited, split into about 900,000 rows each, and gzipped
I have tried several methods. Currently I am using pandas to process the CSVs, everything else is written in Python. It just feels a little too slow.
What’s the fastest method?
shell-script awk python csv csvkit
asked 51 mins ago
Jedi_mcJedi_mc
1
New contributor
Jedi_mc is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
I have some large CSV that need to be processed. I need them pipe delimited, split into about 900,000 rows each, and gzipped
I have tried several methods. Currently I am using pandas to process the CSVs, everything else is written in Python. It just feels a little too slow.
What’s the fastest method?
shell-script awk python csv csvkit
shell-script awk python csv csvkit
asked 51 mins ago
Jedi_mcJedi_mc
1
New contributor
Jedi_mc is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked 51 mins ago
Jedi_mcJedi_mc
1
New contributor
Jedi_mc is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked 51 mins ago
Jedi_mcJedi_mc
1
New contributor
Jedi_mc is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked 51 mins ago
Jedi_mcJedi_mc
1
asked 51 mins ago
Jedi_mcJedi_mc
1
1
New contributor
Jedi_mc is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
New contributor
Jedi_mc is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
add a comment
|
add a comment
|
0
active
oldest
votes
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "106"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/4.0/"u003ecc by-sa 4.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Jedi_mc is a new contributor. Be nice, and check out our Code of Conduct.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f544714%2fpiping-splitting-and-zipping-a-csv-in-linux%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Jedi_mc is a new contributor. Be nice, and check out our Code of Conduct.
Jedi_mc is a new contributor. Be nice, and check out our Code of Conduct.
Jedi_mc is a new contributor. Be nice, and check out our Code of Conduct.
Jedi_mc is a new contributor. Be nice, and check out our Code of Conduct.
Thanks for contributing an answer to Unix & Linux Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f544714%2fpiping-splitting-and-zipping-a-csv-in-linux%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
