Replace text in big file using sedstring substituton for match patternDiffing two big text filesReplace some...
How does a simple logistic regression model achieve a 92% classification accuracy on MNIST?
Ambiguity in notation resolved by +
Were Roman public roads build by private companies?
Why do sellers care about down payments?
How to prepare for STEM graduate program while in industry?
Why some files are not movable in Windows 10
What organs or modifications would be needed for a life biological creature not to require sleep?
Does my opponent need to prove his creature has morph?
Mutable named tuple with default value and conditional rounding support
What are uses of the byte after BRK instruction on 6502?
I was promised a work PC but still awaiting approval 3 months later so using my own laptop - Is it fair to ask employer for laptop insurance?
Can I tap all my opponent's lands while they're casting a spell to negate it?
What is the mathematical notation for rounding a given number to the nearest integer?
How do certain apps show new notifications when internet access is restricted to them?
Speedometer as a symbol into awesomebox
How can I discourage sharing internal API keys within a company?
What officially disallows US presidents from driving?
"Literally" Vs "In the true sense of the word"
Can derivatives be defined as anti-integrals?
In what state are satellites left in when they are left in a graveyard orbit?
Is the Dodge action perceptible to other characters?
Output a Super Mario Image
Write a function that returns an iterable object of all valid points 4-directionally adjacent to (x, y)
How to stabilise the bicycle seatpost and saddle when it is all the way up?
Replace text in big file using sed
string substituton for match patternDiffing two big text filesReplace some text with sedReplace a range of text with special characters using sedSed: replace text keeping digit occurrenceReplace text quickly in very large fileNot able to replace text using sedreplace/remove text (sed) by a fileReplace *n.*n= using sed
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ margin-bottom:0;
}
I have a big file that contains a text will be replaced based on a regular expression passed to.
The regular expression is tested and can find matching pattern, but when I use it with sed
no text replaced.
Note that my environment is Windows, and I have 16GB memory, file size is ~14GB.
sed -i "/,[rn]+ CONSTRAINT `[a-zA-Z0-9_]+` FOREIGN KEY (`[a-zA-Z0-9_]+`) REFERENCES `[a-zA-Z0-9_]+` (`[a-zA-Z0-9_]+`)/ s// /g" all_files_test.sql
all_files_test.sql
file:
DROP TABLE IF EXISTS `holdings_FLZWHX`;
;
;
CREATE TABLE `holdings_FLZWHX` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`DateAdded` datetime NOT NULL,
`FundId` int(11) NOT NULL,
`AssetId` int(11) NOT NULL,
`DayChangeEqt` decimal(18,2) DEFAULT NULL,
`PMinDayRet` decimal(18,2) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `holdings_FLZWHX_FundId` (`FundId`),
KEY `holdings_FLZWHX_AssetId` (`AssetId`),
KEY `holdings_FLZWHX_TypeId` (`TypeId`),
KEY `holdings_FLZWHX_TickerId` (`TickerId`),
KEY `holdings_FLZWHX_StyleId` (`StyleId`),
KEY `holdings_FLZWHX_SectorId` (`SectorId`),
KEY `holdings_FLZWHX_CountryId` (`CountryId`),
KEY `holdings_FLZWHX_CurrencyId` (`CurrencyId`),
CONSTRAINT `holdings_FLZWHX_ibfk_1` FOREIGN KEY (`FundId`) REFERENCES `target_funds` (`fundid`),
CONSTRAINT `holdings_FLZWHX_ibfk_2` FOREIGN KEY (`AssetId`) REFERENCES `asset` (`id`),
CONSTRAINT `holdings_FLZWHX_ibfk_3` FOREIGN KEY (`TypeId`) REFERENCES `type` (`id`),
CONSTRAINT `holdings_FLZWHX_ibfk_4` FOREIGN KEY (`TickerId`) REFERENCES `ticker` (`id`),
CONSTRAINT `holdings_FLZWHX_ibfk_5` FOREIGN KEY (`StyleId`) REFERENCES `style` (`id`),
CONSTRAINT `holdings_FLZWHX_ibfk_6` FOREIGN KEY (`SectorId`) REFERENCES `sector` (`id`),
CONSTRAINT `holdings_FLZWHX_ibfk_7` FOREIGN KEY (`CountryId`) REFERENCES `country` (`id`),
CONSTRAINT `holdings_FLZWHX_ibfk_8` FOREIGN KEY (`CurrencyId`) REFERENCES `currency` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=4201 DEFAULT CHARSET=latin1;
;
The result I was expecting:
DROP TABLE IF EXISTS `holdings_FLZWHX`;
;
;
CREATE TABLE `holdings_FLZWHX` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`DateAdded` datetime NOT NULL,
`FundId` int(11) NOT NULL,
`AssetId` int(11) NOT NULL,
`DayChangeEqt` decimal(18,2) DEFAULT NULL,
`PMinDayRet` decimal(18,2) DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=4201 DEFAULT CHARSET=latin1;
;
sed regular-expression large-files
New contributor
|
show 2 more comments
I have a big file that contains a text will be replaced based on a regular expression passed to.
The regular expression is tested and can find matching pattern, but when I use it with sed
no text replaced.
Note that my environment is Windows, and I have 16GB memory, file size is ~14GB.
sed -i "/,[rn]+ CONSTRAINT `[a-zA-Z0-9_]+` FOREIGN KEY (`[a-zA-Z0-9_]+`) REFERENCES `[a-zA-Z0-9_]+` (`[a-zA-Z0-9_]+`)/ s// /g" all_files_test.sql
all_files_test.sql
file:
DROP TABLE IF EXISTS `holdings_FLZWHX`;
;
;
CREATE TABLE `holdings_FLZWHX` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`DateAdded` datetime NOT NULL,
`FundId` int(11) NOT NULL,
`AssetId` int(11) NOT NULL,
`DayChangeEqt` decimal(18,2) DEFAULT NULL,
`PMinDayRet` decimal(18,2) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `holdings_FLZWHX_FundId` (`FundId`),
KEY `holdings_FLZWHX_AssetId` (`AssetId`),
KEY `holdings_FLZWHX_TypeId` (`TypeId`),
KEY `holdings_FLZWHX_TickerId` (`TickerId`),
KEY `holdings_FLZWHX_StyleId` (`StyleId`),
KEY `holdings_FLZWHX_SectorId` (`SectorId`),
KEY `holdings_FLZWHX_CountryId` (`CountryId`),
KEY `holdings_FLZWHX_CurrencyId` (`CurrencyId`),
CONSTRAINT `holdings_FLZWHX_ibfk_1` FOREIGN KEY (`FundId`) REFERENCES `target_funds` (`fundid`),
CONSTRAINT `holdings_FLZWHX_ibfk_2` FOREIGN KEY (`AssetId`) REFERENCES `asset` (`id`),
CONSTRAINT `holdings_FLZWHX_ibfk_3` FOREIGN KEY (`TypeId`) REFERENCES `type` (`id`),
CONSTRAINT `holdings_FLZWHX_ibfk_4` FOREIGN KEY (`TickerId`) REFERENCES `ticker` (`id`),
CONSTRAINT `holdings_FLZWHX_ibfk_5` FOREIGN KEY (`StyleId`) REFERENCES `style` (`id`),
CONSTRAINT `holdings_FLZWHX_ibfk_6` FOREIGN KEY (`SectorId`) REFERENCES `sector` (`id`),
CONSTRAINT `holdings_FLZWHX_ibfk_7` FOREIGN KEY (`CountryId`) REFERENCES `country` (`id`),
CONSTRAINT `holdings_FLZWHX_ibfk_8` FOREIGN KEY (`CurrencyId`) REFERENCES `currency` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=4201 DEFAULT CHARSET=latin1;
;
The result I was expecting:
DROP TABLE IF EXISTS `holdings_FLZWHX`;
;
;
CREATE TABLE `holdings_FLZWHX` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`DateAdded` datetime NOT NULL,
`FundId` int(11) NOT NULL,
`AssetId` int(11) NOT NULL,
`DayChangeEqt` decimal(18,2) DEFAULT NULL,
`PMinDayRet` decimal(18,2) DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=4201 DEFAULT CHARSET=latin1;
;
sed regular-expression large-files
New contributor
@msp9011 I added the expected result.
– H Aßdøµ
26 mins ago
trysed '/CONSTRAINT/d;/ KEY/d' all_files_test.sql
– msp9011
21 mins ago
unix.stackexchange.com/questions/532869/… Check this link. It's because of the quotes.
– rahuls36
19 mins ago
But I need the full pattern I set.
– H Aßdøµ
14 mins ago
Note that[rn]+
matches on either a backslashr
orn
followed by a+
.sed
regexps are basic regexps by default. In any case,sed
works on one line at a time so what it matches on will never contain a newline character.
– Stéphane Chazelas
3 mins ago
|
show 2 more comments
I have a big file that contains a text will be replaced based on a regular expression passed to.
The regular expression is tested and can find matching pattern, but when I use it with sed
no text replaced.
Note that my environment is Windows, and I have 16GB memory, file size is ~14GB.
sed -i "/,[rn]+ CONSTRAINT `[a-zA-Z0-9_]+` FOREIGN KEY (`[a-zA-Z0-9_]+`) REFERENCES `[a-zA-Z0-9_]+` (`[a-zA-Z0-9_]+`)/ s// /g" all_files_test.sql
all_files_test.sql
file:
DROP TABLE IF EXISTS `holdings_FLZWHX`;
;
;
CREATE TABLE `holdings_FLZWHX` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`DateAdded` datetime NOT NULL,
`FundId` int(11) NOT NULL,
`AssetId` int(11) NOT NULL,
`DayChangeEqt` decimal(18,2) DEFAULT NULL,
`PMinDayRet` decimal(18,2) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `holdings_FLZWHX_FundId` (`FundId`),
KEY `holdings_FLZWHX_AssetId` (`AssetId`),
KEY `holdings_FLZWHX_TypeId` (`TypeId`),
KEY `holdings_FLZWHX_TickerId` (`TickerId`),
KEY `holdings_FLZWHX_StyleId` (`StyleId`),
KEY `holdings_FLZWHX_SectorId` (`SectorId`),
KEY `holdings_FLZWHX_CountryId` (`CountryId`),
KEY `holdings_FLZWHX_CurrencyId` (`CurrencyId`),
CONSTRAINT `holdings_FLZWHX_ibfk_1` FOREIGN KEY (`FundId`) REFERENCES `target_funds` (`fundid`),
CONSTRAINT `holdings_FLZWHX_ibfk_2` FOREIGN KEY (`AssetId`) REFERENCES `asset` (`id`),
CONSTRAINT `holdings_FLZWHX_ibfk_3` FOREIGN KEY (`TypeId`) REFERENCES `type` (`id`),
CONSTRAINT `holdings_FLZWHX_ibfk_4` FOREIGN KEY (`TickerId`) REFERENCES `ticker` (`id`),
CONSTRAINT `holdings_FLZWHX_ibfk_5` FOREIGN KEY (`StyleId`) REFERENCES `style` (`id`),
CONSTRAINT `holdings_FLZWHX_ibfk_6` FOREIGN KEY (`SectorId`) REFERENCES `sector` (`id`),
CONSTRAINT `holdings_FLZWHX_ibfk_7` FOREIGN KEY (`CountryId`) REFERENCES `country` (`id`),
CONSTRAINT `holdings_FLZWHX_ibfk_8` FOREIGN KEY (`CurrencyId`) REFERENCES `currency` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=4201 DEFAULT CHARSET=latin1;
;
The result I was expecting:
DROP TABLE IF EXISTS `holdings_FLZWHX`;
;
;
CREATE TABLE `holdings_FLZWHX` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`DateAdded` datetime NOT NULL,
`FundId` int(11) NOT NULL,
`AssetId` int(11) NOT NULL,
`DayChangeEqt` decimal(18,2) DEFAULT NULL,
`PMinDayRet` decimal(18,2) DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=4201 DEFAULT CHARSET=latin1;
;
sed regular-expression large-files
New contributor
I have a big file that contains a text will be replaced based on a regular expression passed to.
The regular expression is tested and can find matching pattern, but when I use it with sed
no text replaced.
Note that my environment is Windows, and I have 16GB memory, file size is ~14GB.
sed -i "/,[rn]+ CONSTRAINT `[a-zA-Z0-9_]+` FOREIGN KEY (`[a-zA-Z0-9_]+`) REFERENCES `[a-zA-Z0-9_]+` (`[a-zA-Z0-9_]+`)/ s// /g" all_files_test.sql
all_files_test.sql
file:
DROP TABLE IF EXISTS `holdings_FLZWHX`;
;
;
CREATE TABLE `holdings_FLZWHX` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`DateAdded` datetime NOT NULL,
`FundId` int(11) NOT NULL,
`AssetId` int(11) NOT NULL,
`DayChangeEqt` decimal(18,2) DEFAULT NULL,
`PMinDayRet` decimal(18,2) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `holdings_FLZWHX_FundId` (`FundId`),
KEY `holdings_FLZWHX_AssetId` (`AssetId`),
KEY `holdings_FLZWHX_TypeId` (`TypeId`),
KEY `holdings_FLZWHX_TickerId` (`TickerId`),
KEY `holdings_FLZWHX_StyleId` (`StyleId`),
KEY `holdings_FLZWHX_SectorId` (`SectorId`),
KEY `holdings_FLZWHX_CountryId` (`CountryId`),
KEY `holdings_FLZWHX_CurrencyId` (`CurrencyId`),
CONSTRAINT `holdings_FLZWHX_ibfk_1` FOREIGN KEY (`FundId`) REFERENCES `target_funds` (`fundid`),
CONSTRAINT `holdings_FLZWHX_ibfk_2` FOREIGN KEY (`AssetId`) REFERENCES `asset` (`id`),
CONSTRAINT `holdings_FLZWHX_ibfk_3` FOREIGN KEY (`TypeId`) REFERENCES `type` (`id`),
CONSTRAINT `holdings_FLZWHX_ibfk_4` FOREIGN KEY (`TickerId`) REFERENCES `ticker` (`id`),
CONSTRAINT `holdings_FLZWHX_ibfk_5` FOREIGN KEY (`StyleId`) REFERENCES `style` (`id`),
CONSTRAINT `holdings_FLZWHX_ibfk_6` FOREIGN KEY (`SectorId`) REFERENCES `sector` (`id`),
CONSTRAINT `holdings_FLZWHX_ibfk_7` FOREIGN KEY (`CountryId`) REFERENCES `country` (`id`),
CONSTRAINT `holdings_FLZWHX_ibfk_8` FOREIGN KEY (`CurrencyId`) REFERENCES `currency` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=4201 DEFAULT CHARSET=latin1;
;
The result I was expecting:
DROP TABLE IF EXISTS `holdings_FLZWHX`;
;
;
CREATE TABLE `holdings_FLZWHX` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`DateAdded` datetime NOT NULL,
`FundId` int(11) NOT NULL,
`AssetId` int(11) NOT NULL,
`DayChangeEqt` decimal(18,2) DEFAULT NULL,
`PMinDayRet` decimal(18,2) DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=4201 DEFAULT CHARSET=latin1;
;
sed regular-expression large-files
sed regular-expression large-files
New contributor
New contributor
edited 26 mins ago
H Aßdøµ
New contributor
asked 42 mins ago
H AßdøµH Aßdøµ
1013 bronze badges
1013 bronze badges
New contributor
New contributor
@msp9011 I added the expected result.
– H Aßdøµ
26 mins ago
trysed '/CONSTRAINT/d;/ KEY/d' all_files_test.sql
– msp9011
21 mins ago
unix.stackexchange.com/questions/532869/… Check this link. It's because of the quotes.
– rahuls36
19 mins ago
But I need the full pattern I set.
– H Aßdøµ
14 mins ago
Note that[rn]+
matches on either a backslashr
orn
followed by a+
.sed
regexps are basic regexps by default. In any case,sed
works on one line at a time so what it matches on will never contain a newline character.
– Stéphane Chazelas
3 mins ago
|
show 2 more comments
@msp9011 I added the expected result.
– H Aßdøµ
26 mins ago
trysed '/CONSTRAINT/d;/ KEY/d' all_files_test.sql
– msp9011
21 mins ago
unix.stackexchange.com/questions/532869/… Check this link. It's because of the quotes.
– rahuls36
19 mins ago
But I need the full pattern I set.
– H Aßdøµ
14 mins ago
Note that[rn]+
matches on either a backslashr
orn
followed by a+
.sed
regexps are basic regexps by default. In any case,sed
works on one line at a time so what it matches on will never contain a newline character.
– Stéphane Chazelas
3 mins ago
@msp9011 I added the expected result.
– H Aßdøµ
26 mins ago
@msp9011 I added the expected result.
– H Aßdøµ
26 mins ago
try
sed '/CONSTRAINT/d;/ KEY/d' all_files_test.sql
– msp9011
21 mins ago
try
sed '/CONSTRAINT/d;/ KEY/d' all_files_test.sql
– msp9011
21 mins ago
unix.stackexchange.com/questions/532869/… Check this link. It's because of the quotes.
– rahuls36
19 mins ago
unix.stackexchange.com/questions/532869/… Check this link. It's because of the quotes.
– rahuls36
19 mins ago
But I need the full pattern I set.
– H Aßdøµ
14 mins ago
But I need the full pattern I set.
– H Aßdøµ
14 mins ago
Note that
[rn]+
matches on either a backslash r
or n
followed by a +
. sed
regexps are basic regexps by default. In any case, sed
works on one line at a time so what it matches on will never contain a newline character.– Stéphane Chazelas
3 mins ago
Note that
[rn]+
matches on either a backslash r
or n
followed by a +
. sed
regexps are basic regexps by default. In any case, sed
works on one line at a time so what it matches on will never contain a newline character.– Stéphane Chazelas
3 mins ago
|
show 2 more comments
0
active
oldest
votes
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "106"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/4.0/"u003ecc by-sa 4.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
H Aßdøµ is a new contributor. Be nice, and check out our Code of Conduct.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f541516%2freplace-text-in-big-file-using-sed%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
H Aßdøµ is a new contributor. Be nice, and check out our Code of Conduct.
H Aßdøµ is a new contributor. Be nice, and check out our Code of Conduct.
H Aßdøµ is a new contributor. Be nice, and check out our Code of Conduct.
H Aßdøµ is a new contributor. Be nice, and check out our Code of Conduct.
Thanks for contributing an answer to Unix & Linux Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f541516%2freplace-text-in-big-file-using-sed%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
@msp9011 I added the expected result.
– H Aßdøµ
26 mins ago
try
sed '/CONSTRAINT/d;/ KEY/d' all_files_test.sql
– msp9011
21 mins ago
unix.stackexchange.com/questions/532869/… Check this link. It's because of the quotes.
– rahuls36
19 mins ago
But I need the full pattern I set.
– H Aßdøµ
14 mins ago
Note that
[rn]+
matches on either a backslashr
orn
followed by a+
.sed
regexps are basic regexps by default. In any case,sed
works on one line at a time so what it matches on will never contain a newline character.– Stéphane Chazelas
3 mins ago