Mdthbs

German equivalent to "going down the rabbit hole"

What are the in-game differences between WoW Classic and the original 2006 Version

Create a list of snaking numbers under 50,000

Eliminate key lookup in execution plan

LWC: Is it safe to rely on window.location.href to get the page url?

'spazieren' - walking in a silly and affected manner?

Can authors email you PDFs of their textbook for free?

How can I improve my formal definitions

I was given someone else's visa, stamped in my passport

Moscow SVO airport, how to avoid scam taxis without pre-booking?

Small RAM 4 KB on the early Apple II?

Cheap oscilloscope showing 16 MHz square wave

Why don't 3D printer heads use ceramic inner walls?

Can two aircraft be allowed to stay on the same runway at the same time?

How do I portray irrational anger in first person?

Can inductive kick be discharged without freewheeling diode, in this example?

What checks exist against overuse of presidential pardons in the USA?

Under GDPR, can I give permission once to allow everyone to store and process my data?

What are ways to record who took the pictures if a camera is used by multiple people?

Who declared the Last Alliance to be the "last" and why?

What is a "hashed transaction" in SQL Server Replication terminology?

Is it good practice to speed up and slow down where not written in a song?

How can I portray a character with no fear of death, without them sounding utterly bored?

Can I leave a large suitcase at TPE during a 4-hour layover, and pick it up 4.5 days later when I come back to TPE on my way to Taipei downtown?

How can I decompress and print the last few lines of a compressed text file?

Can I compress a compressed file more?Mount compressed tar file and update itHow can I tell when a file was compressed?Unpack file compressed in .txz and .tar with one commandIs there a compression method that supports solid compression and also adding data to the compressed file?Proper way to create a compressed, rsyncable mirror of a directory?Delete all lines that contain duplicate letters

.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ margin-bottom:0;
}

3

I have 6 gzipped text files, each of which is ~17G when compressed. I need to see the last few lines (decompressed) of each file to check whether a particular problem is there. The obvious approach is very slow:

for i in *; do zcat "$i" | tail -n3; done

I was thinking I could do something clever like:

for i in *; do tail -n 30 "$i" | gunzip | tail -n 4 ; done

Or

for i in *; do tac "$i" | head -100 | gunzip | tac | tail -n3; done

But both complain about:

gzip: stdin: not in gzip format

I thought that was because I was missing the gzip header, but this also fails:

$ aa=$(head -c 300 file.gz)

$ bb=$(tail -c 300 file.gz)

$ printf '%s%s' "$aa" "$bb" | gunzip

gzip: stdin: unexpected end of file

What I am really looking for is a ztail or ztac but I don't think those exist. Can anyone come up with a clever trick that lets me decompress and print the last few lines of a compressed file without decompressing the entire thing?

shell command-line compression

share|improve this question

asked Jun 28 '16 at 10:17

terdon♦terdon

142k3434 gold badges292292 silver badges470470 bronze badges

• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  
  I don't think you can... See also How can I tail a zipped file without reading its entire contents? and the links there...
  
  – don_crissti
  Jun 28 '16 at 10:23
  
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  @don_crissti well, dang! Looks like you're right. I'll leave this for a while in case anyone comes up with something really clever, and if not I'll copy one of the SO answers over. Probably the one stating that gzip doesn't support this. Feel free to do it yourself, by the way.
  
  – terdon♦
  Jun 28 '16 at 10:27
  

  
  
  
  

add a comment | 

3

I have 6 gzipped text files, each of which is ~17G when compressed. I need to see the last few lines (decompressed) of each file to check whether a particular problem is there. The obvious approach is very slow:

for i in *; do zcat "$i" | tail -n3; done

I was thinking I could do something clever like:

for i in *; do tail -n 30 "$i" | gunzip | tail -n 4 ; done

Or

for i in *; do tac "$i" | head -100 | gunzip | tac | tail -n3; done

But both complain about:

gzip: stdin: not in gzip format

I thought that was because I was missing the gzip header, but this also fails:

$ aa=$(head -c 300 file.gz)

$ bb=$(tail -c 300 file.gz)

$ printf '%s%s' "$aa" "$bb" | gunzip

gzip: stdin: unexpected end of file

What I am really looking for is a ztail or ztac but I don't think those exist. Can anyone come up with a clever trick that lets me decompress and print the last few lines of a compressed file without decompressing the entire thing?

shell command-line compression

share|improve this question

asked Jun 28 '16 at 10:17

terdon♦terdon

142k3434 gold badges292292 silver badges470470 bronze badges

• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  
  I don't think you can... See also How can I tail a zipped file without reading its entire contents? and the links there...
  
  – don_crissti
  Jun 28 '16 at 10:23
  
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  @don_crissti well, dang! Looks like you're right. I'll leave this for a while in case anyone comes up with something really clever, and if not I'll copy one of the SO answers over. Probably the one stating that gzip doesn't support this. Feel free to do it yourself, by the way.
  
  – terdon♦
  Jun 28 '16 at 10:27
  

  
  
  
  

add a comment | 

I have 6 gzipped text files, each of which is ~17G when compressed. I need to see the last few lines (decompressed) of each file to check whether a particular problem is there. The obvious approach is very slow:

for i in *; do zcat "$i" | tail -n3; done

I was thinking I could do something clever like:

for i in *; do tail -n 30 "$i" | gunzip | tail -n 4 ; done

Or

for i in *; do tac "$i" | head -100 | gunzip | tac | tail -n3; done

But both complain about:

gzip: stdin: not in gzip format

I thought that was because I was missing the gzip header, but this also fails:

$ aa=$(head -c 300 file.gz)

$ bb=$(tail -c 300 file.gz)

$ printf '%s%s' "$aa" "$bb" | gunzip

gzip: stdin: unexpected end of file

What I am really looking for is a ztail or ztac but I don't think those exist. Can anyone come up with a clever trick that lets me decompress and print the last few lines of a compressed file without decompressing the entire thing?

shell command-line compression

share|improve this question

asked Jun 28 '16 at 10:17

terdon♦terdon

142k3434 gold badges292292 silver badges470470 bronze badges

for i in *; do zcat "$i" | tail -n3; done

for i in *; do tail -n 30 "$i" | gunzip | tail -n 4 ; done

for i in *; do tac "$i" | head -100 | gunzip | tac | tail -n3; done

gzip: stdin: not in gzip format

$ aa=$(head -c 300 file.gz)

$ bb=$(tail -c 300 file.gz)

$ printf '%s%s' "$aa" "$bb" | gunzip

gzip: stdin: unexpected end of file

share|improve this question

asked Jun 28 '16 at 10:17

terdon♦terdon

142k3434 gold badges292292 silver badges470470 bronze badges

share|improve this question

asked Jun 28 '16 at 10:17

terdon♦terdon

142k3434 gold badges292292 silver badges470470 bronze badges

asked Jun 28 '16 at 10:17

terdon♦terdon

142k3434 gold badges292292 silver badges470470 bronze badges

asked Jun 28 '16 at 10:17

terdon♦terdon

142k3434 gold badges292292 silver badges470470 bronze badges

2 Answers
2

active

oldest

votes

5

You can't, as it has been already said, if the files have been compressed with standard gzip. If you have control over the compression, you can use dictzip to compress the files, it compresses the files in separate blocks and you can decompress just the last block (typically 64KB). And it is backward compatible with gzip, meaning the dictzipped file is perfectly legal gzipped file as well.

Other possibility would be if you get the gzipped file as a concatenation of several already gzipped files, you could search for the last gzip signature and decompress everything after that.

share|improve this answer

edited May 23 '17 at 12:39

Community♦

1

answered Jun 28 '16 at 15:04

Radovan GarabíkRadovan Garabík

1,44866 silver badges1212 bronze badges

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  I don't, sadly. I was hoping I could use the idea here to get the right chunks but I don't know if that's possible or what the chunks might be.
  
  – terdon♦
  Jun 28 '16 at 15:05
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  @terdon The idea there is to create the compressed file in chunks and then only uncompress the last chunk. If the compressed file has a single chunk, which is the case if you produced it with gzip, then “only uncompress the last chunk” = “uncompress the sole chunk” = “uncompress the whole file”.
  
  – Gilles
  Jun 28 '16 at 21:29
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  @Gilles I see. I had understood that the "chunks" were a result of the gzip algorithm and could perhaps be inferred from the compressed file.
  
  – terdon♦
  Jun 29 '16 at 8:54
  

  
  
  
  

add a comment | 

0

Well, you can access randomly a gzipped file if you previously creates an index for each file ...

I've develop the command line tool that you were probably looking for: it can access the tail using the same amount of time than a gunzip... BUT because it creates a little (<<1%/gzip) index, next extractions will be really quick:

https://github.com/circulosmeos/gztool

The tool has two options that may be of interest for you:

• 
  -S option supervise a still-growing file and creates an index for it as it is growing - this can be useful for gzipped rsyslog files as reduces to zero in the practice the time of index creation.


• 
  -t tails a gzip file: this way you can do: $ gztool -t foo.gz
  Please, note that if the index doesn't exists, this will consume the same time as a complete decompression: but as the index is reusable, next searches will be greatly reduced in time - and as it is the same time, why not use it and create the index at the same time?

This tool is based on zran.c demonstration code from original zlib, so there's no out-of-the-rules magic!

share|improve this answer

answered 3 hours ago

circulosmeoscirculosmeos

14111 gold badge11 silver badge44 bronze badges

add a comment | 

Your Answer

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "106"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f292556%2fhow-can-i-decompress-and-print-the-last-few-lines-of-a-compressed-text-file%23new-answer', 'question_page');
}
);

Post as a guest

Name

Email

Required, but never shown

5

You can't, as it has been already said, if the files have been compressed with standard gzip. If you have control over the compression, you can use dictzip to compress the files, it compresses the files in separate blocks and you can decompress just the last block (typically 64KB). And it is backward compatible with gzip, meaning the dictzipped file is perfectly legal gzipped file as well.

Other possibility would be if you get the gzipped file as a concatenation of several already gzipped files, you could search for the last gzip signature and decompress everything after that.

share|improve this answer

edited May 23 '17 at 12:39

Community♦

1

answered Jun 28 '16 at 15:04

Radovan GarabíkRadovan Garabík

1,44866 silver badges1212 bronze badges

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  I don't, sadly. I was hoping I could use the idea here to get the right chunks but I don't know if that's possible or what the chunks might be.
  
  – terdon♦
  Jun 28 '16 at 15:05
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  @terdon The idea there is to create the compressed file in chunks and then only uncompress the last chunk. If the compressed file has a single chunk, which is the case if you produced it with gzip, then “only uncompress the last chunk” = “uncompress the sole chunk” = “uncompress the whole file”.
  
  – Gilles
  Jun 28 '16 at 21:29
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  @Gilles I see. I had understood that the "chunks" were a result of the gzip algorithm and could perhaps be inferred from the compressed file.
  
  – terdon♦
  Jun 29 '16 at 8:54
  

  
  
  
  

add a comment | 

5

You can't, as it has been already said, if the files have been compressed with standard gzip. If you have control over the compression, you can use dictzip to compress the files, it compresses the files in separate blocks and you can decompress just the last block (typically 64KB). And it is backward compatible with gzip, meaning the dictzipped file is perfectly legal gzipped file as well.

Other possibility would be if you get the gzipped file as a concatenation of several already gzipped files, you could search for the last gzip signature and decompress everything after that.

share|improve this answer

edited May 23 '17 at 12:39

Community♦

1

answered Jun 28 '16 at 15:04

Radovan GarabíkRadovan Garabík

1,44866 silver badges1212 bronze badges

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  I don't, sadly. I was hoping I could use the idea here to get the right chunks but I don't know if that's possible or what the chunks might be.
  
  – terdon♦
  Jun 28 '16 at 15:05
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  @terdon The idea there is to create the compressed file in chunks and then only uncompress the last chunk. If the compressed file has a single chunk, which is the case if you produced it with gzip, then “only uncompress the last chunk” = “uncompress the sole chunk” = “uncompress the whole file”.
  
  – Gilles
  Jun 28 '16 at 21:29
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  @Gilles I see. I had understood that the "chunks" were a result of the gzip algorithm and could perhaps be inferred from the compressed file.
  
  – terdon♦
  Jun 29 '16 at 8:54
  

  
  
  
  

add a comment | 

You can't, as it has been already said, if the files have been compressed with standard gzip. If you have control over the compression, you can use dictzip to compress the files, it compresses the files in separate blocks and you can decompress just the last block (typically 64KB). And it is backward compatible with gzip, meaning the dictzipped file is perfectly legal gzipped file as well.

Other possibility would be if you get the gzipped file as a concatenation of several already gzipped files, you could search for the last gzip signature and decompress everything after that.

share|improve this answer

edited May 23 '17 at 12:39

Community♦

1

answered Jun 28 '16 at 15:04

Radovan GarabíkRadovan Garabík

1,44866 silver badges1212 bronze badges

share|improve this answer

edited May 23 '17 at 12:39

Community♦

1

answered Jun 28 '16 at 15:04

Radovan GarabíkRadovan Garabík

1,44866 silver badges1212 bronze badges

edited May 23 '17 at 12:39

Community♦

1

edited May 23 '17 at 12:39

Community♦

1

answered Jun 28 '16 at 15:04

Radovan GarabíkRadovan Garabík

1,44866 silver badges1212 bronze badges

answered Jun 28 '16 at 15:04

Radovan GarabíkRadovan Garabík

1,44866 silver badges1212 bronze badges

0

Well, you can access randomly a gzipped file if you previously creates an index for each file ...

I've develop the command line tool that you were probably looking for: it can access the tail using the same amount of time than a gunzip... BUT because it creates a little (<<1%/gzip) index, next extractions will be really quick:

https://github.com/circulosmeos/gztool

The tool has two options that may be of interest for you:

• 
  -S option supervise a still-growing file and creates an index for it as it is growing - this can be useful for gzipped rsyslog files as reduces to zero in the practice the time of index creation.


• 
  -t tails a gzip file: this way you can do: $ gztool -t foo.gz
  Please, note that if the index doesn't exists, this will consume the same time as a complete decompression: but as the index is reusable, next searches will be greatly reduced in time - and as it is the same time, why not use it and create the index at the same time?

This tool is based on zran.c demonstration code from original zlib, so there's no out-of-the-rules magic!

share|improve this answer

answered 3 hours ago

circulosmeoscirculosmeos

14111 gold badge11 silver badge44 bronze badges

add a comment | 

0

Well, you can access randomly a gzipped file if you previously creates an index for each file ...

I've develop the command line tool that you were probably looking for: it can access the tail using the same amount of time than a gunzip... BUT because it creates a little (<<1%/gzip) index, next extractions will be really quick:

https://github.com/circulosmeos/gztool

The tool has two options that may be of interest for you:

• 
  -S option supervise a still-growing file and creates an index for it as it is growing - this can be useful for gzipped rsyslog files as reduces to zero in the practice the time of index creation.


• 
  -t tails a gzip file: this way you can do: $ gztool -t foo.gz
  Please, note that if the index doesn't exists, this will consume the same time as a complete decompression: but as the index is reusable, next searches will be greatly reduced in time - and as it is the same time, why not use it and create the index at the same time?

This tool is based on zran.c demonstration code from original zlib, so there's no out-of-the-rules magic!

share|improve this answer

answered 3 hours ago

circulosmeoscirculosmeos

14111 gold badge11 silver badge44 bronze badges

add a comment | 

Well, you can access randomly a gzipped file if you previously creates an index for each file ...

I've develop the command line tool that you were probably looking for: it can access the tail using the same amount of time than a gunzip... BUT because it creates a little (<<1%/gzip) index, next extractions will be really quick:

https://github.com/circulosmeos/gztool

The tool has two options that may be of interest for you:

• 
  -S option supervise a still-growing file and creates an index for it as it is growing - this can be useful for gzipped rsyslog files as reduces to zero in the practice the time of index creation.


• 
  -t tails a gzip file: this way you can do: $ gztool -t foo.gz
  Please, note that if the index doesn't exists, this will consume the same time as a complete decompression: but as the index is reusable, next searches will be greatly reduced in time - and as it is the same time, why not use it and create the index at the same time?

This tool is based on zran.c demonstration code from original zlib, so there's no out-of-the-rules magic!

share|improve this answer

answered 3 hours ago

circulosmeoscirculosmeos

14111 gold badge11 silver badge44 bronze badges

share|improve this answer

answered 3 hours ago

circulosmeoscirculosmeos

14111 gold badge11 silver badge44 bronze badges

answered 3 hours ago

circulosmeoscirculosmeos

14111 gold badge11 silver badge44 bronze badges

answered 3 hours ago

circulosmeoscirculosmeos

14111 gold badge11 silver badge44 bronze badges